Nov 03 23:25:40.178538 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 23:25:40.178566 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 23:25:40.178574 ip-10-0-143-134 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 23:25:40.178793 ip-10-0-143-134 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 23:25:50.245078 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 23:25:50.245097 ip-10-0-143-134 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 363ad058b8064a50b5d1117eb7f8a500 --
Nov 03 23:28:31.116705 ip-10-0-143-134 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 23:28:31.539331 ip-10-0-143-134 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:31.539331 ip-10-0-143-134 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 23:28:31.539331 ip-10-0-143-134 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:31.539331 ip-10-0-143-134 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547030    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547119    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547125    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547130    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547135    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547140    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.547133 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547144    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547148    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547153    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547156    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547160    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547163    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547167    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547171    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547174    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547178    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547181    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547185    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547189    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547193    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547197    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547201    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547204    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547209    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547213    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.547550 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547216    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547220    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547224    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547228    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547232    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547236    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547240    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547244    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547248    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547252    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547255    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547259    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547263    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547267    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547271    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547276    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547280    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547285    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547292    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547300    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.548314 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547304    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547309    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547314    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547318    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547322    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547326    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547330    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547335    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547339    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547344    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547348    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547352    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547357    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547361    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547365    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547369    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547374    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547378    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547383    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547401    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.548945 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547406    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547410    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547414    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547422    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547430    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547435    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547442    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547447    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547451    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547455    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547460    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547464    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547468    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547473    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547477    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547482    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547487    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547491    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547495    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.549425 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547499    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547504    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547508    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547513    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547517    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547522    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547526    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547530    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547535    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547539    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547544    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547548    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547552    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547556    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547561    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547565    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547569    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547574    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547578    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.549954 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547582    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547589    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.547593    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550527    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550540    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550547    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550552    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550557    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550561    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550566    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550570    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550574    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550579    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550583    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550588    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550592    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550596    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550600    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550605    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550609    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.550621 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550613    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550617    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550621    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550625    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550629    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550633    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550637    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550641    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550645    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550649    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550654    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550658    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550662    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550666    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550675    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550680    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550684    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550688    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550693    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550697    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.551457 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550701    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550705    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550709    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550713    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550718    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550722    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550726    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550730    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550737    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550743    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550748    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550753    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550757    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550761    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550767    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550771    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550776    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550780    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550786    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.552225 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550790    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550795    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550799    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550803    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550807    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550811    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550816    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550820    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550825    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550830    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550834    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550838    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550842    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550846    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550851    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550857    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550863    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550868    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550873    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.552919 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550877    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550881    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550885    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550890    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550894    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550898    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550903    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550907    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550912    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550917    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550921    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550925    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550930    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550934    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550938    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550942    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550946    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550951    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550955    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550959    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.553364 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550963    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550968    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550972    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550977    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550980    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550985    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550989    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550995    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.550999    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.551003    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.551007    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553089    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553103    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553107    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553110    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553113    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553116    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553119    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553122    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553124    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.553893 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553127    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553130    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553132    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553135    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553138    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553140    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553143    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553146    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553149    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553151    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553154    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553157    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553160    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553162    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553165    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553168    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553170    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553173    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553175    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553178    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.554382 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553180    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553184    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553186    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553190    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553193    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553196    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553199    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553202    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553204    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553207    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553210    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553212    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553215    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553217    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553220    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553222    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553226    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553230    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553233    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.554866 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553236    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553238    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553242    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553244    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553247    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553250    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553253    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553255    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553258    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553261    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553264    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553266    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553269    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553271    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553274    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553277    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553280    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553283    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553286    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553288    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.555339 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553291    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553294    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553297    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553299    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553302    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553304    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553307    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553309    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553312    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553314    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553317    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553319    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553322    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553324    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553328    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553331    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553334    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553336    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553339    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.555828 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553342    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553345    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553348    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553350    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553353    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553355    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553358    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553362    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553365    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553369    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553372    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553375    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553378    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553381    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553383    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553404    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553407    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553411    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.553414    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554368    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 23:28:31.556268 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554378    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554399    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554406    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554413    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554417    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554421    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554426    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554430    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554433    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554439    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554442    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554446    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554449    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554452    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554455    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554458    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554461    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554465    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554468    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554471    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554475    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554479    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554483    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 23:28:31.556782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554486    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554489    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554493    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554496    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554499    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554502    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554505    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554508    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554513    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554515    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554519    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554521    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554525    2574 flags.go:64] FLAG: --enable-server="true"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554528    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554532    2574 flags.go:64] FLAG: --event-burst="100"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554536    2574 flags.go:64] FLAG: --event-qps="50"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554539    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554543    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554546    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554551    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554555    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554558    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554561    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554564    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554567    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 23:28:31.557308 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554571    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554574    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554576    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554579    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554582    2574 flags.go:64] FLAG: --feature-gates=""
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554586    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554590    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554593    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554596    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554599    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554602    2574 flags.go:64] FLAG: --help="false"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554605    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554609    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554612    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554615    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554618    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554621    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554625    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554627    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554630    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554633    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554636    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554640    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554643    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 03 23:28:31.557982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554646    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554649    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554652    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554655    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554658    2574 flags.go:64] FLAG: --lock-file=""
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554661    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554664    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554667    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554673    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554676    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554679    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554682    2574 flags.go:64] FLAG: --logging-format="text"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554685    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554688    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554692    2574 flags.go:64] FLAG: --manifest-url=""
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554695    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554700    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554703    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554707    2574 flags.go:64] FLAG: --max-pods="110"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554710    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554713    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554716    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554720    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554724    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554727    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 23:28:31.558679 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554730    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554738    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554741    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554744    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554748    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554751    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554754    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554757    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554760    2574 flags.go:64] FLAG: --port="10250"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554763    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554766    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04a95d504492b9e3a"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554770    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554773    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554776    2574 flags.go:64] FLAG: --register-node="true"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554779    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554783    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554786    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554789    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554792    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554795    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554799    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554804    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554807    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554810    2574 flags.go:64] FLAG: --runonce="false"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554813    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554816    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554820    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 23:28:31.559271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554823    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554828    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554832    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554835    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554838    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554842    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554845    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554848    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554851    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554854    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554857    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554860    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554865    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554868    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554871    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554877    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554880    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554884    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554887    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554890    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554893    2574 flags.go:64] FLAG: --v="2"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554898    2574 flags.go:64] FLAG: --version="false"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554903    2574 flags.go:64] FLAG: --vmodule=""
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554907    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.554910    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 23:28:31.559928 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555044    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555048    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555051    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555054    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555058    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555062    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555065    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555068    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555072    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555075    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555078    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555080    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555083    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555086    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555088    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555091    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555095    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555099    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555102    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.560539 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555105    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555108    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555110    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555113    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555116    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555118    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555122    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555124    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555127    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555130    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555132    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555135    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555137    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555140    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555143    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555145    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555148    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555150    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555153    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555155    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.561028 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555158    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555162    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555165    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555167    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555170    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555172    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555175    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555178    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555180    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555183    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555185    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555187    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555190    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555193    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555195    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555198    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555201    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555203    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555206    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.561578 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555210    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555212    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555215    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555217    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555220    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555223    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555226    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555228    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555231    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555234    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555236    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555239    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555241    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555244    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555248    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555250    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555253    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555256    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555258    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555261    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.562032 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555264    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555266    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555269    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555271    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555274    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555277    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555279    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555282    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555284    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555287    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555290    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555292    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555295    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555298    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555301    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555303    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555306    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555308    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555311    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.562546 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555314    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555317    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555320    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555322    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555325    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555327    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555329    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555334    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555336    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.555364    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555426    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555432    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555435    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555438    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555441    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.562990 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555444    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555446    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555449    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555451    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555454    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555457    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555459    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555462    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555464    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555469    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555472    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555474    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555478    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555480    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555483    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555485    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555488    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555492    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555495    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555498    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.563358 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555500    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555504    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555508    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555510    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555513    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555516    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555519    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555522    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555525    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555528    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555530    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555533    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555535    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555538    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555540    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555543    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555546    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555548    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555551    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555553    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.563863 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555556    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555558    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555561    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555563    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555566    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555569    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555571    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555574    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555576    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555581    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555583    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555586    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555588    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555591    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555593    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555596    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555598    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555601    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555604    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555607    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.564346 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555609    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555612    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555614    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555617    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555619    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555622    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555624    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555627    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555629    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555632    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555634    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555637    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555639    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555642    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555644    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555648    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555651    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555653    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555656    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555659    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.564838 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555661    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555665    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555668    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555670    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555673    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555675    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555678    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555680    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555683    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555686    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555688    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555691    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555693    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555696    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555699    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555701    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555704    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555707    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555709    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.565312 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555712    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.555714    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.562883    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.562898    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.562917    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.562922    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.562983    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.562989    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.562992    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.562995    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.562998    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563002    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563004    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563007    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563010    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563013    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563016    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563018    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563021    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563024    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.565776 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563027    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563029    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563032    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563036    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563040    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563043    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563047    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563049    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563052    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563055    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563058    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563061    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563064    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563067    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563069    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563072    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563075    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563078    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.566252 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563081    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563083    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563086    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563088    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563091    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563093    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563096    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563099    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563102    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563104    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563107    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563110    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563113    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563115    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563118    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563120    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563123    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563125    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563128    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.566789 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563130    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563133    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563136    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563138    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563141    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563144    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563147    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563150    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563152    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563155    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563158    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563160    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563163    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563165    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563168    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563170    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563173    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563176    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563178    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563181    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.567285 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563184    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563187    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563190    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563193    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563195    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563198    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563201    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563203    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563206    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563209    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563211    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563214    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563217    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563220    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563222    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563225    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563228    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563231    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563233    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563236    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.567800 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563238    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563241    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563244    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563247    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563249    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563252    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563254    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563257    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563260    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563262    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563265    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563267    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563270    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563273    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563277    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.563305    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:31.568258 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563490    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563496    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563500    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563503    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563506    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563510    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563513    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563516    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563519    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563522    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563525    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563528    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563530    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563533    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563536    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563538    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563541    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563544    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563547    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563550    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.568660 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563553    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563555    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563558    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563560    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563563    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563565    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563568    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563570    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563573    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563576    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563579    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563582    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563584    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563587    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563590    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563593    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563597    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563600    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563603    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.569174 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563605    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563608    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563611    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563613    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563616    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563618    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563621    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563624    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563627    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563631    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563634    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563637    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563640    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563643    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563645    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563648    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563650    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563653    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563656    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.569670 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563658    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563661    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563663    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563665    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563668    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563671    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563674    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563676    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563678    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563681    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563684    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563686    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563689    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563691    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563694    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563697    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563700    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563703    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563705    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563708    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.570126 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563710    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563713    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563716    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563718    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563721    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563723    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563726    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563729    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563731    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563733    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563736    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563738    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563741    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563744    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563746    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563749    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563751    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563754    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563757    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563760    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.570651 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563762    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563764    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563767    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563770    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563772    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563775    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563777    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563780    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.563804    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563841    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563845    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563848    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563852    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563855    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563857    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563860    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:28:31.571222 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563863    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563866    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563869    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563872    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563875    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563877    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563880    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563883    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563885    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563888    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563890    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563893    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563896    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563898    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563901    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563904    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563907    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563909    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563912    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563914    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:28:31.571614 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563917    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563919    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563922    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563925    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563927    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563930    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563932    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563935    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563937    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563940    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563942    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563945    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563947    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563950    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563952    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563955    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563957    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563960    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563962    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563965    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:28:31.572074 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563967    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563969    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563972    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563975    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563977    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563980    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563982    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563985    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563988    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563991    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563993    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563996    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.563998    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564001    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564003    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564006    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564009    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564012    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564014    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564017    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:28:31.572555 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564019    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564022    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564024    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564027    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564030    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564032    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564035    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564037    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564039    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564042    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564044    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564047    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564049    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564052    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564054    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564057    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564059    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564062    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564065    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:28:31.573136 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564067    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564070    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564073    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564076    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564080    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564083    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564086    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564089    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564092    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564094    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564097    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564099    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564102    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564104    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564107    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564109    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564113    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564119    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:28:31.573603 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564122    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:31.564125    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.564179    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.566264    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.567229    2574 server.go:1014] "Starting client certificate rotation"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.567334    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.567382    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.569056    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:31.574056 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.570595    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:28:31.582203 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.582182    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 03 23:28:31.585316 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.585301    2574 log.go:25] "Validated CRI v1 image API"
Nov 03 23:28:31.588711 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.588688    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 23:28:31.592729 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.592708    2574 fs.go:135] Filesystem UUIDs: map[2551f046-6da1-4d05-b790-453b79579363:/dev/nvme0n1p3 5cb435c4-7145-4bfb-b970-a05d5c3bc160:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 23:28:31.592793 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.592729    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 23:28:31.597961 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.597843    2574 manager.go:217] Machine: {Timestamp:2026-11-03 23:28:31.596408896 +0000 UTC m=+0.355611961 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100451 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd3738a6e17481780ecdeb6b200df SystemUUID:ec2fd373-8a6e-1748-1780-ecdeb6b200df BootID:363ad058-b806-4a50-b5d1-117eb7f8a500 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:14:fe:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:14:fe:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:88:6f:cc:bd:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 23:28:31.598043 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.597971    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 23:28:31.598080 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.598059    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 23:28:31.601525 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.601496    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 23:28:31.601666 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.601527    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-134.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 23:28:31.601707 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.601670    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 23:28:31.601707 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.601678    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 23:28:31.601707 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.601701    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:31.602229 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.602212    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 23:28:31.602398 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.602375    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:28:31.603216 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.603205    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 23:28:31.606813 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.606802    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 23:28:31.606951 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.606942    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 23:28:31.606984 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.606976    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 23:28:31.607010 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.606990    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 23:28:31.607010 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.607009    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 23:28:31.607061 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.607022    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 03 23:28:31.607061 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.607033    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 23:28:31.607587 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.607573    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:31.609865 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.609850    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 23:28:31.615537 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.615524    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:28:31.616293 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616282    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 23:28:31.616327 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616311    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 23:28:31.616501 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616491    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 23:28:31.616538 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616507    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 23:28:31.616538 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616514    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 23:28:31.616538 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616519    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 23:28:31.616538 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616525    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 23:28:31.616538 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.616531    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 23:28:31.617074 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617066    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 23:28:31.617101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617075    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 23:28:31.617101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617082    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 23:28:31.617101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617088    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 23:28:31.617101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617097    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 23:28:31.617222 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.617105    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 23:28:31.618439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.618426    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 23:28:31.618475 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.618444    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 23:28:31.622270 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.622257    2574 server.go:1263] "Started kubelet"
Nov 03 23:28:31.622602 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.622335    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 23:28:31.622602 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.622409    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 23:28:31.622602 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.622502    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 23:28:31.623090 ip-10-0-143-134 systemd[1]: Started Kubernetes Kubelet.
Nov 03 23:28:31.624262 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.624246    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 23:28:31.628893 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.628872    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 23:28:31.629707 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.629687    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 23:28:31.631101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.631073    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-134.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 23:28:31.631814 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.631797    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 23:28:31.633000 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.632986    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 23:28:31.633118 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.633093    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.633551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633533    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 23:28:31.633646 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633535    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 23:28:31.633646 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633568    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 23:28:31.633722 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633656    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 23:28:31.633722 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633662    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 23:28:31.633938 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633923    2574 factory.go:55] Registering systemd factory
Nov 03 23:28:31.633938 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.633940    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 03 23:28:31.634184 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.634170    2574 factory.go:153] Registering CRI-O factory
Nov 03 23:28:31.634223 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.634187    2574 factory.go:223] Registration of the crio container factory successfully
Nov 03 23:28:31.634252 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.634226    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 23:28:31.634252 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.634242    2574 factory.go:103] Registering Raw factory
Nov 03 23:28:31.634328 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.634254    2574 manager.go:1199] Started watching for new ooms in manager
Nov 03 23:28:31.635122 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.635110    2574 manager.go:319] Starting recovery of all containers
Nov 03 23:28:31.636648 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.636628    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpfxn"
Nov 03 23:28:31.644207 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.644191    2574 manager.go:324] Recovery completed
Nov 03 23:28:31.644295 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.644273    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 23:28:31.644366 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.644345    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-134.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 23:28:31.644555 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.644519    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-134.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 23:28:31.645497 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.644492    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.us-east-2.compute.internal.18e4acc483a0e8d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:31.622228184 +0000 UTC m=+0.381431248,LastTimestamp:2026-11-03 23:28:31.622228184 +0000 UTC m=+0.381431248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.us-east-2.compute.internal,}"
Nov 03 23:28:31.647317 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.647300    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xpfxn"
Nov 03 23:28:31.647457 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.647342    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 23:28:31.649959 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.649947    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:31.651653 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.651635    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:31.651742 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.651667    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:31.651742 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.651678    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:31.652232 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652216    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 03 23:28:31.652232 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652229    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 23:28:31.652370 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652242    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:31.652370 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652248    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 23:28:31.652953 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652935    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:31.653037 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652964    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:31.653037 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.652977    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:31.654632 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.654615    2574 policy_none.go:50] "Start"
Nov 03 23:28:31.654704 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.654637    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 23:28:31.654704 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.654648    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 23:28:31.655368 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.654191    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.us-east-2.compute.internal.18e4acc48561e1f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-134.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:31.651652088 +0000 UTC m=+0.410855153,LastTimestamp:2026-11-03 23:28:31.651652088 +0000 UTC m=+0.410855153,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.us-east-2.compute.internal,}"
Nov 03 23:28:31.657611 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.657591    2574 policy_none.go:44] "Start"
Nov 03 23:28:31.733668 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.733628    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.834213 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.834051    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.839969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.839934    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 23:28:31.842279 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842266    2574 manager.go:341] "Starting Device Plugin manager"
Nov 03 23:28:31.842337 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.842294    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 23:28:31.842337 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842304    2574 server.go:87] "Starting device plugin registration server"
Nov 03 23:28:31.842618 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842605    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 23:28:31.842690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842619    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 23:28:31.842765 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842738    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 23:28:31.842829 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842819    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 23:28:31.842881 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.842829    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 23:28:31.843424 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.843376    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 23:28:31.843515 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.843446    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.849007 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.848989    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-134.us-east-2.compute.internal\" not found" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:31.943439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.943410    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:31.944459 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.944444    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:31.944539 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.944478    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:31.944539 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.944493    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:31.944539 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.944524    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:31.952928 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.952914    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:31.952975 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.952939    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-134.us-east-2.compute.internal\": node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.964059 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.964027    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 23:28:31.965445 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.965418    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 23:28:31.965445 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.965445    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 23:28:31.965588 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.965466    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 23:28:31.965588 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.965500    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 23:28:31.970321 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:31.970299    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:31.974152 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:31.974134    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:32.066632 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.066554    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"]
Nov 03 23:28:32.066632 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.066632    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:32.068459 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.068442    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:32.068550 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.068472    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:32.068550 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.068482    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:32.070564 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.070550    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:32.070654 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.070586    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.070742 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.070724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.070796 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.070767    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:32.071344 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071326    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:32.071439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071359    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:32.071439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071369    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:32.071439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071326    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:32.071439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071433    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:32.071583 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.071445    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:32.073431 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.073417    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.073486 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.073446    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:28:32.074306 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.074291    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:28:32.074362 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.074320    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:28:32.074362 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.074331    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:28:32.101911 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.101889    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.104610 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.104592    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.136488 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.136463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.136572 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.136491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.136572 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.136511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.171373 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.171346    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.236660 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.236725 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.236725 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.236794 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236754    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.236830 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236755    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.236830 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.236755    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.271798 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.271765    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.372251 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.372224    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.405423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.405368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.407526 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.407512    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:32.473100 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.473060    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.567474 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.567442    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 23:28:32.567896 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.567602    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 23:28:32.573696 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.573671    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.651902 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.651816    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:31 +0000 UTC" deadline="2028-08-15 21:15:35.68256276 +0000 UTC"
Nov 03 23:28:32.651902 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.651854    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15621h47m3.030710628s"
Nov 03 23:28:32.674688 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.674661    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.775068 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.775043    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.875537 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.875507    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.885529 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.885504    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:32.923336 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.923274    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:32.972031 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:32.972001    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a44b6090d9fc279a15fb8a9a158ae7.slice/crio-104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa WatchSource:0}: Error finding container 104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa: Status 404 returned error can't find the container with id 104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa
Nov 03 23:28:32.975584 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:32.975563    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:28:32.980725 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:32.980368    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:28:32.988680 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:32.988656    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03308d9024657dde1775c95b6ad40488.slice/crio-66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124 WatchSource:0}: Error finding container 66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124: Status 404 returned error can't find the container with id 66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124
Nov 03 23:28:33.073649 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.073621    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:33.133724 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.133693    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:33.144588 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.144572    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:33.145467 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.145454    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:28:33.153172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.153159    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:28:33.608238 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.607961    2574 apiserver.go:52] "Watching apiserver"
Nov 03 23:28:33.613766 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.613476    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 23:28:33.614465 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.614433    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lz9rf","openshift-network-operator/iptables-alerter-wf22x","kube-system/global-pull-secret-syncer-fpnrk","kube-system/konnectivity-agent-jnzl8","openshift-multus/multus-additional-cni-plugins-hm9dt","openshift-cluster-node-tuning-operator/tuned-ffxlw","openshift-dns/node-resolver-784mp","openshift-network-diagnostics/network-check-target-zr4h4","openshift-ovn-kubernetes/ovnkube-node-qts9p","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg","openshift-image-registry/node-ca-6rhdg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal","openshift-multus/multus-llrvd"]
Nov 03 23:28:33.616838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.616822    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.618909 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.618889    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.620067 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.620047    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.620161 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.620103    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.620161 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.620154    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\""
Nov 03 23:28:33.620981 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.620960    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.621315 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.621297    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 23:28:33.621448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.621435    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.621499 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.621475    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.621534 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.621519    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\""
Nov 03 23:28:33.623353 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.623630 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623611    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.623709 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623702    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 23:28:33.623765 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623755    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.623765 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623760    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 23:28:33.623838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.623773    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\""
Nov 03 23:28:33.626058 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.626039    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 23:28:33.626155 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.626047    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\""
Nov 03 23:28:33.626155 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.626105    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 23:28:33.626155 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.626145    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.628271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.628253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.628819 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.628800    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 23:28:33.628819 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.628813    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 23:28:33.629409 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.628956    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\""
Nov 03 23:28:33.630935 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.630916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.632265 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.632250    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 23:28:33.633361 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.633347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:33.633452 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.633435    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:33.634701 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.634682    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.635610 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.635591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.637891 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.637876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.640524 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.640226    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.640524 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.640301    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:33.642745 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.642837 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.642837 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.642837 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.642837 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.642806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:33.642837 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.642993 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.642993 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.643054 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.642998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.643054 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.643054 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.643151 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpsp\" (UniqueName: \"kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.643151 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.643151 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.643151 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.643291 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643230    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.643291 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.643291 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.643447 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.643447 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.643516 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.643594 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643512    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 23:28:33.643594 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.643594 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:33.643594 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.643838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.643838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.643838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.643838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.643838 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.644314 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.643801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwzf\" (UniqueName: \"kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.644379 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.644379 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.644507 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.644507 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9bw\" (UniqueName: \"kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.644575 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644518    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.644575 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.644690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644605    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.644690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.644690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.644690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6pt\" (UniqueName: \"kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.644988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.645462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.644990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.645462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.645462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.645462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.645462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgpzz\" (UniqueName: \"kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645703    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.645746 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645943    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89x9\" (UniqueName: \"kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.645976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.646003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.646031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.646089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.646117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.646172 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.646151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.667737 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.667713    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2nbj"
Nov 03 23:28:33.678209 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.678189    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2nbj"
Nov 03 23:28:33.720782 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.720721    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 23:28:33.721078 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721054    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721083    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721082    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721045    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721147    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721164    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721169    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721180    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721172    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\""
Nov 03 23:28:33.721195 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721171    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721232    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721248    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721300    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721322    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721356    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 23:28:33.721574 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.721361    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 23:28:33.734606 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.734581    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 23:28:33.747039 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747187 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747187 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6pt\" (UniqueName: \"kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747300 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747300 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747204    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747300 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.747300 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747290    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747368    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747437    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.747518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747520    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747539    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747596    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgpzz\" (UniqueName: \"kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747630    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747787    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.747926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rjq\" (UniqueName: \"kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.747995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89x9\" (UniqueName: \"kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.748770 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748418    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpsp\" (UniqueName: \"kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748683    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748696    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748770    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.749622 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748799    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.748904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749125    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749161    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749174    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749235    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749252    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749319    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749316    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749484    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.750376 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749585    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749624    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749647    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749692    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749734    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749739    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749864    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749934    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.749992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750036    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750074    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750099    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750106    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750145    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750148    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751111 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750231    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750917    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.750983    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751052    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751256    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751312    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751444    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.751969 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751541    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwzf\" (UniqueName: \"kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.751889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7v7\" (UniqueName: \"kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9bw\" (UniqueName: \"kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752445    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752624    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.752744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.752917    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.753075    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.753448 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.753171    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.753644 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.753485    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.756210 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.756185    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.756355 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.756295    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.756545 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.756527    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.758105 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.758072    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.759007 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.758988    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6pt\" (UniqueName: \"kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.760187 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.760168    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgpzz\" (UniqueName: \"kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.763081 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.763057    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:33.763172 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.763085    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:33.763172 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.763099    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:33.763172 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.763170    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:34.2631496 +0000 UTC m=+3.022352686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:33.763343 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.763301    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89x9\" (UniqueName: \"kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.763688 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.763667    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.763833 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.763814    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpsp\" (UniqueName: \"kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.764020 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.764003    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9bw\" (UniqueName: \"kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.767941 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.767918    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwzf\" (UniqueName: \"kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.853405 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.853551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rjq\" (UniqueName: \"kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.853551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.853551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.853551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853482    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.853551 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853544    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.853757 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.853757 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.853757 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7v7\" (UniqueName: \"kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.853757 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.853757 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.853738    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:33.853987 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.853768    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:33.853987 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.853820    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:34.353784402 +0000 UTC m=+3.112987471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:33.853987 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853776    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:33.853987 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:33.853835    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:34.353828663 +0000 UTC m=+3.113031715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:33.853987 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.853927    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.865986 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.865908    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7v7\" (UniqueName: \"kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.865986 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.865923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rjq\" (UniqueName: \"kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:33.929484 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.929450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-784mp"
Nov 03 23:28:33.940689 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.940654    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04000032_9127_46a5_ba82_dcbb72b1e394.slice/crio-18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d WatchSource:0}: Error finding container 18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d: Status 404 returned error can't find the container with id 18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d
Nov 03 23:28:33.941940 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.941914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:28:33.946067 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.946039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llrvd"
Nov 03 23:28:33.952487 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.952461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:33.953474 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.953448    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e628017_63f5_4329_86b0_f07e84304f96.slice/crio-ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375 WatchSource:0}: Error finding container ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375: Status 404 returned error can't find the container with id ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375
Nov 03 23:28:33.956940 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.956868    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeb5df1_2d70_4963_ab3f_503518320882.slice/crio-2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966 WatchSource:0}: Error finding container 2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966: Status 404 returned error can't find the container with id 2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966
Nov 03 23:28:33.963283 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.963252    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ddeb79_97e0_492a_a4d6_f50f96720fb7.slice/crio-4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c WatchSource:0}: Error finding container 4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c: Status 404 returned error can't find the container with id 4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c
Nov 03 23:28:33.971774 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.971714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:28:33.973618 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.973575    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnzl8" event={"ID":"b0ddeb79-97e0-492a-a4d6-f50f96720fb7","Type":"ContainerStarted","Data":"4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c"}
Nov 03 23:28:33.973761 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.973740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:28:33.975264 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.975235    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llrvd" event={"ID":"5eeb5df1-2d70-4963-ab3f-503518320882","Type":"ContainerStarted","Data":"2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966"}
Nov 03 23:28:33.976518 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.976497    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg" event={"ID":"0e628017-63f5-4329-86b0-f07e84304f96","Type":"ContainerStarted","Data":"ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375"}
Nov 03 23:28:33.977644 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.977622    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-784mp" event={"ID":"04000032-9127-46a5-ba82-dcbb72b1e394","Type":"ContainerStarted","Data":"18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d"}
Nov 03 23:28:33.978776 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.978755    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerStarted","Data":"66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124"}
Nov 03 23:28:33.980126 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.980107    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"b6a44b6090d9fc279a15fb8a9a158ae7","Type":"ContainerStarted","Data":"104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa"}
Nov 03 23:28:33.984841 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.984795    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a32217f_0822_43ae_8541_066002fca229.slice/crio-8727f0bf4529f443774676a22afd54929e5ca5813efac071a776ecdd565f5042 WatchSource:0}: Error finding container 8727f0bf4529f443774676a22afd54929e5ca5813efac071a776ecdd565f5042: Status 404 returned error can't find the container with id 8727f0bf4529f443774676a22afd54929e5ca5813efac071a776ecdd565f5042
Nov 03 23:28:33.986605 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.986539    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b2245e_79d5_4117_a5e1_2dfc4ee0e5c3.slice/crio-70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719 WatchSource:0}: Error finding container 70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719: Status 404 returned error can't find the container with id 70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719
Nov 03 23:28:33.989470 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.989451    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:28:33.992191 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.992173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:28:33.997276 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:33.997256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:33.999255 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:33.999236    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9570168c_b9aa_455f_981e_33d57c4b5671.slice/crio-90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284 WatchSource:0}: Error finding container 90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284: Status 404 returned error can't find the container with id 90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284
Nov 03 23:28:34.002841 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:34.002812    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b695b8_84da_4df4_87f7_c3aa1ccdcf9f.slice/crio-09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c WatchSource:0}: Error finding container 09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c: Status 404 returned error can't find the container with id 09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c
Nov 03 23:28:34.009560 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:28:34.009530    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17332ae_befe_4495_8904_af7ad229d464.slice/crio-53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0 WatchSource:0}: Error finding container 53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0: Status 404 returned error can't find the container with id 53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0
Nov 03 23:28:34.136178 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.136099    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:28:34.357916 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.357877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:34.358110 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358047    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:34.358110 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358073    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:34.358110 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358086    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:34.358289 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358147    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:35.358127408 +0000 UTC m=+4.117330462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:34.358289 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.358183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:34.358289 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.358216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:34.358464 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358338    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:34.358464 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358370    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:35.358359935 +0000 UTC m=+4.117563000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:34.358784 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358768    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:34.358835 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.358817    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:35.358802476 +0000 UTC m=+4.118005542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:34.681330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.681269    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:33 +0000 UTC" deadline="2028-06-23 11:08:49.298824954 +0000 UTC"
Nov 03 23:28:34.681330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.681325    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14339h40m14.617503264s"
Nov 03 23:28:34.965812 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.965734    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:34.965985 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:34.965874    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:34.986462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.986354    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6rhdg" event={"ID":"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f","Type":"ContainerStarted","Data":"09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c"}
Nov 03 23:28:34.988200 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.988131    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wf22x" event={"ID":"9570168c-b9aa-455f-981e-33d57c4b5671","Type":"ContainerStarted","Data":"90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284"}
Nov 03 23:28:34.991584 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.991558    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerStarted","Data":"70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719"}
Nov 03 23:28:34.995185 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.995162    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ffxlw" event={"ID":"5a32217f-0822-43ae-8541-066002fca229","Type":"ContainerStarted","Data":"8727f0bf4529f443774676a22afd54929e5ca5813efac071a776ecdd565f5042"}
Nov 03 23:28:34.999634 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:34.999607    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0"}
Nov 03 23:28:35.367703 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.367616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:35.367703 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.367671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:35.367703 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.367701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:35.368006 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.367822    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:35.368006 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.367883    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.367865609 +0000 UTC m=+6.127068667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368335    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368360    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368373    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368439    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.36842423 +0000 UTC m=+6.127627302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368509    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:35.368591 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.368550    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.368537103 +0000 UTC m=+6.127740167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:35.681611 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.681538    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:33 +0000 UTC" deadline="2028-08-21 02:00:11.539921066 +0000 UTC"
Nov 03 23:28:35.681611 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.681572    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15746h31m35.85835328s"
Nov 03 23:28:35.972620 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.972541    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:35.972773 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.972682    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:35.973102 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:35.973077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:35.973223 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:35.973178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:36.010091 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:36.009568    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"b6a44b6090d9fc279a15fb8a9a158ae7","Type":"ContainerStarted","Data":"abd2cb92d1b9d1ee277a186477de1fbd27ffbe24400759db5fa02bb4dc099b34"}
Nov 03 23:28:36.024728 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:36.024661    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" podStartSLOduration=3.024641867 podStartE2EDuration="3.024641867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:36.024051002 +0000 UTC m=+4.783254077" watchObservedRunningTime="2026-11-03 23:28:36.024641867 +0000 UTC m=+4.783844944"
Nov 03 23:28:36.966191 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:36.966004    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:36.966670 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:36.966290    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:37.014601 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.014563    2574 generic.go:356] "Generic (PLEG): container finished" podID="03308d9024657dde1775c95b6ad40488" containerID="a37043bc03df2d1a7c7c695cd19da0cf398dfccd66022d143e633a3a93390627" exitCode=0
Nov 03 23:28:37.014788 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.014656    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerDied","Data":"a37043bc03df2d1a7c7c695cd19da0cf398dfccd66022d143e633a3a93390627"}
Nov 03 23:28:37.021755 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.021723    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="765c29ea950379321c39c246afee21b2edec2f6ed642b6867f4f72463ad0f166" exitCode=0
Nov 03 23:28:37.022695 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.022671    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"765c29ea950379321c39c246afee21b2edec2f6ed642b6867f4f72463ad0f166"}
Nov 03 23:28:37.388423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.388307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:37.388423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.388361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:37.388423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.388411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:37.388756 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.388571    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:37.388756 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.388635    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:41.388617812 +0000 UTC m=+10.147820871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:37.389082 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389061    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:37.389178 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389086    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:37.389178 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389099    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:37.389178 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389146    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:41.389132227 +0000 UTC m=+10.148335292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:37.389372 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389211    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:37.389372 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.389248    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:41.389234503 +0000 UTC m=+10.148437559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:37.969234 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.968918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:37.969234 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.969034    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:37.969724 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:37.969351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:37.969724 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:37.969452    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:38.036211 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:38.036172    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerStarted","Data":"dc48f0316b9105fc766822ae687c743672026411c497c28036affeb086913953"}
Nov 03 23:28:38.966224 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:38.966136    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:38.966407 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:38.966270    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:39.042439 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.041802    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6rhdg" event={"ID":"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f","Type":"ContainerStarted","Data":"9b3fb792534c7bc86a33908e1a3f6ddc5bb613a86bb4155135dfb81dd6ce31f3"}
Nov 03 23:28:39.045534 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.045503    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg" event={"ID":"0e628017-63f5-4329-86b0-f07e84304f96","Type":"ContainerStarted","Data":"52c6a35cf23e510d3364c66546a9e172bb4b56ae44e69d5ecdbed98afb9b0c57"}
Nov 03 23:28:39.060723 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.060664    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6rhdg" podStartSLOduration=4.134320334 podStartE2EDuration="8.060643793s" totalImagesPullingTime="3.926323459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:39.059910677 +0000 UTC m=+7.819113754" watchObservedRunningTime="2026-11-03 23:28:39.060643793 +0000 UTC m=+7.819846872"
Nov 03 23:28:39.060884 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.060788    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" podStartSLOduration=6.060778222 podStartE2EDuration="6.060778222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:38.056500314 +0000 UTC m=+6.815703389" watchObservedRunningTime="2026-11-03 23:28:39.060778222 +0000 UTC m=+7.819981296"
Nov 03 23:28:39.972028 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.972001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:39.972028 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.972019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:39.972463 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:39.972122    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:39.972463 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:39.972203    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:39.995072 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:39.995039    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 23:28:40.048594 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.048557    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnzl8" event={"ID":"b0ddeb79-97e0-492a-a4d6-f50f96720fb7","Type":"ContainerStarted","Data":"3b61556c79f2ffb3f012b051da9093ec75b4313386bb2b27f5304e35031680a8"}
Nov 03 23:28:40.052018 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.051750    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg" event={"ID":"0e628017-63f5-4329-86b0-f07e84304f96","Type":"ContainerStarted","Data":"ad78ab1db7a4348dd4472f74098c5ea82b01f98b159df34a740743ad899b84a2"}
Nov 03 23:28:40.065064 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.065023    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jnzl8" podStartSLOduration=3.39443882 podStartE2EDuration="9.065008567s" totalImagesPullingTime="5.670569747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:40.064284394 +0000 UTC m=+8.823487482" watchObservedRunningTime="2026-11-03 23:28:40.065008567 +0000 UTC m=+8.824211640"
Nov 03 23:28:40.848401 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.848259    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:28:39.995058923Z","UUID":"4fefd177-49e7-4e59-a39e-b23ba1b11749","Handler":null,"Name":"","Endpoint":""}
Nov 03 23:28:40.850075 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.850053    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 23:28:40.850075 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.850085    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 23:28:40.966368 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:40.966332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:40.966560 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:40.966486    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:41.055614 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.055584    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llrvd" event={"ID":"5eeb5df1-2d70-4963-ab3f-503518320882","Type":"ContainerStarted","Data":"5a2ca8b03121b957c6aaedc443e5414888d651228163fcdaafd57ccc63bfbbfd"}
Nov 03 23:28:41.431580 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.431540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.431583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.431616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431725    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431730    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431756    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431772    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431784    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:49.431766618 +0000 UTC m=+18.190969688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431819    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:49.431804344 +0000 UTC m=+18.191007409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431727    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:41.431847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.431882    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:49.431867177 +0000 UTC m=+18.191070232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:41.966847 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.966648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:41.967034 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:41.966719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:41.967034 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.966916    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:41.967131 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:41.967031    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:42.060483 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.060444    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg" event={"ID":"0e628017-63f5-4329-86b0-f07e84304f96","Type":"ContainerStarted","Data":"4ef25800009231b0c833e8f13c27e110f9c53581f924811e301aab44ae1f8e6b"}
Nov 03 23:28:42.062626 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.062597    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wf22x" event={"ID":"9570168c-b9aa-455f-981e-33d57c4b5671","Type":"ContainerStarted","Data":"467a52161446da29d822e2a5dc391f9dbf8ba44cabe2cb8b8cfb50eaabbbf0f3"}
Nov 03 23:28:42.083062 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.083012    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-llrvd" podStartSLOduration=5.038925056 podStartE2EDuration="11.083000042s" totalImagesPullingTime="6.044074986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:41.078002162 +0000 UTC m=+9.837205335" watchObservedRunningTime="2026-11-03 23:28:42.083000042 +0000 UTC m=+10.842203114"
Nov 03 23:28:42.083735 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.083702    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg" podStartSLOduration=4.320534829 podStartE2EDuration="11.083690631s" totalImagesPullingTime="6.763155802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:42.082766234 +0000 UTC m=+10.841969308" watchObservedRunningTime="2026-11-03 23:28:42.083690631 +0000 UTC m=+10.842893704"
Nov 03 23:28:42.098869 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.098826    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wf22x" podStartSLOduration=4.738532552 podStartE2EDuration="11.098812953s" totalImagesPullingTime="6.360280401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:42.098667506 +0000 UTC m=+10.857870579" watchObservedRunningTime="2026-11-03 23:28:42.098812953 +0000 UTC m=+10.858016026"
Nov 03 23:28:42.816582 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.816509    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:42.817090 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.817069    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:42.966152 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:42.966120    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:42.966298 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:42.966222    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:43.066955 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:43.066870    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="ed68516dd14011c41c31e2bf3eefb53cfe4e53d314b62555c7b3ea90c25b5dfe" exitCode=0
Nov 03 23:28:43.067546 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:43.066989    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"ed68516dd14011c41c31e2bf3eefb53cfe4e53d314b62555c7b3ea90c25b5dfe"}
Nov 03 23:28:43.966178 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:43.965701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:43.966178 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:43.965840    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:43.966178 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:43.965888    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:43.966178 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:43.966020    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:44.071357 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:44.071282    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="06d91eaa1da898ba7b78b280e17029adaf31305bdfe3cb46a1a90783f86f2d6f" exitCode=0
Nov 03 23:28:44.071357 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:44.071328    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"06d91eaa1da898ba7b78b280e17029adaf31305bdfe3cb46a1a90783f86f2d6f"}
Nov 03 23:28:44.966228 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:44.966193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:44.966369 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:44.966302    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:45.965755 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:45.965713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:45.966249 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:45.965722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:45.966249 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:45.965847    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:45.966249 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:45.965912    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:46.079682 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.079064    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="73bc61213f0d2b2ffbbec27d3f33307e19f8ec3a11a1f5db2913cd12de374e35" exitCode=0
Nov 03 23:28:46.079682 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.079120    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"73bc61213f0d2b2ffbbec27d3f33307e19f8ec3a11a1f5db2913cd12de374e35"}
Nov 03 23:28:46.771048 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.770846    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:46.771317 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.771181    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:28:46.772292 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.772263    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:28:46.965694 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:46.965660    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:46.965845 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:46.965783    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:47.966290 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:47.966253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:47.966765 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:47.966361    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:47.966765 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:47.966408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:47.966765 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:47.966476    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:48.086243 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:48.086198    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ffxlw" event={"ID":"5a32217f-0822-43ae-8541-066002fca229","Type":"ContainerStarted","Data":"fbd48d38d071dfda8d54a54906b194f62366bd7604423253da34181f2896a941"}
Nov 03 23:28:48.106358 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:48.106312    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ffxlw" podStartSLOduration=3.116737392 podStartE2EDuration="17.106298313s" totalImagesPullingTime="13.989560921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:48.106164084 +0000 UTC m=+16.865367158" watchObservedRunningTime="2026-11-03 23:28:48.106298313 +0000 UTC m=+16.865501385"
Nov 03 23:28:48.966036 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:48.965997    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:48.966235 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:48.966124    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:49.092200 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.092164    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="2dedb8f986f450475096ca860f5833e5365ad547e779259e4eb3fd19fb439cd6" exitCode=0
Nov 03 23:28:49.092688 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.092259    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"2dedb8f986f450475096ca860f5833e5365ad547e779259e4eb3fd19fb439cd6"}
Nov 03 23:28:49.490345 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.490251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:49.490345 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.490313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.490354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490453    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490479    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490492    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490499    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490520    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490544    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:05.490529431 +0000 UTC m=+34.249732482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490563    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:05.490553242 +0000 UTC m=+34.249756295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:28:49.490585 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.490574    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:05.490569204 +0000 UTC m=+34.249772255 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:28:49.966263 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.966184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:49.966452 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:49.966186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:49.966452 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.966298    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:49.966452 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:49.966378    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:50.098069 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:50.098033    2574 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="6f3cc23eaa70a1bec851e1a831072d395cccce593f90ac80aa5482afbfb28781" exitCode=0
Nov 03 23:28:50.098555 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:50.098081    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerDied","Data":"6f3cc23eaa70a1bec851e1a831072d395cccce593f90ac80aa5482afbfb28781"}
Nov 03 23:28:50.965979 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:50.965940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:50.966155 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:50.966056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:51.104580 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:51.104542    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" event={"ID":"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3","Type":"ContainerStarted","Data":"6b3294cfbca71c67b7b24f8f00f6932ff0e5afe370d14260c7883b7101180271"}
Nov 03 23:28:51.131173 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:51.131122    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hm9dt" podStartSLOduration=8.066686483 podStartE2EDuration="20.131102807s" totalImagesPullingTime="12.064416324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:51.130629225 +0000 UTC m=+19.889832299" watchObservedRunningTime="2026-11-03 23:28:51.131102807 +0000 UTC m=+19.890305912"
Nov 03 23:28:51.967648 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:51.967566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:51.967806 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:51.967664    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:51.967806 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:51.967717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:51.967927 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:51.967857    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:52.108484 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:52.108440    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-784mp" event={"ID":"04000032-9127-46a5-ba82-dcbb72b1e394","Type":"ContainerStarted","Data":"57b4b795f5ebc1088ba7ff895d21b5eea9777749304aa0af32d0116e753cbd30"}
Nov 03 23:28:52.170423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:52.170355    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-784mp" podStartSLOduration=3.472309485 podStartE2EDuration="21.170336974s" totalImagesPullingTime="17.698027489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:52.169573709 +0000 UTC m=+20.928776785" watchObservedRunningTime="2026-11-03 23:28:52.170336974 +0000 UTC m=+20.929540048"
Nov 03 23:28:52.966107 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:52.966080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:52.966259 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:52.966185    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:53.050479 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:53.050453    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-784mp_04000032-9127-46a5-ba82-dcbb72b1e394/dns-node-resolver/0.log"
Nov 03 23:28:53.966895 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:53.966719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:53.967609 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:53.966719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:53.967609 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:53.967003    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:53.967609 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:53.967038    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:54.018285 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.018254    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6rhdg_a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/node-ca/0.log"
Nov 03 23:28:54.118110 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118076    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"8d64779e37a1f3f0d255aeed47f819925de5050454e9c60c38479cb82dfff0b0"}
Nov 03 23:28:54.118312 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118162    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"0c6cf7056c379146f0a1f63b75ab622c59f47c83d9aa52d726d2269d02b56a28"}
Nov 03 23:28:54.118312 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118207    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"0fcf17e25ad89c0c01f56614a7d2a2a1b75cec91208f2f77660829d4cb8e0344"}
Nov 03 23:28:54.118312 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118247    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"d4b07e302c37cf616491007fe3b247166a50e983ecf0168f1be2f5c97745fb7f"}
Nov 03 23:28:54.118312 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118290    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"ff9b087d5f4c828cd13780db4710cc2e7669892450bff183abe56eb3a3eea4cb"}
Nov 03 23:28:54.118496 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.118331    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"565688b53095a4c63fc643a6d03e11a8298126c5b30b38d04f7ea33c000601f4"}
Nov 03 23:28:54.966435 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:54.966382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:54.966693 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:54.966501    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:55.966283 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:55.966248    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:55.966748 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:55.966289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:55.966748 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:55.966374    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:55.966748 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:55.966523    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:56.124806 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:56.124770    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"4e91316d585c7d6233b648767406dc8dd71106a4bd8d8417331f5997c24d8e59"}
Nov 03 23:28:56.965669 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:56.965635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:56.965847 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:56.965736    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:57.965931 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:57.965683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:57.965931 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:57.965713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:57.966516 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:57.965940    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:57.966516 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:57.966006    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:28:58.131951 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:58.131910    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" event={"ID":"c17332ae-befe-4495-8904-af7ad229d464","Type":"ContainerStarted","Data":"27998d84b6476ad691411c1fbc54bc8d5e8a5a739ae0afe2dc9740eff203acf5"}
Nov 03 23:28:58.132186 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:58.132164    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:58.148358 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:58.148332    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:58.180676 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:58.180636    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p" podStartSLOduration=8.015105933 podStartE2EDuration="27.180623563s" totalImagesPullingTime="19.16551763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:58.179208866 +0000 UTC m=+26.938411939" watchObservedRunningTime="2026-11-03 23:28:58.180623563 +0000 UTC m=+26.939826643"
Nov 03 23:28:58.965632 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:58.965603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:28:58.965819 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:58.965761    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:28:59.134363 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:59.134340    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:59.134748 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:59.134427    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:59.149279 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:59.149254    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:28:59.966426 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:59.966369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:28:59.966627 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:28:59.966436    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:28:59.966627 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:59.966508    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:28:59.966627 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:28:59.966611    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:00.965737 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:00.965700    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:00.966156 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:00.965806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:01.966549 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:01.966506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:01.967055 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:01.966618    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:01.967055 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:01.966667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:01.967055 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:01.966762    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:02.966370 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:02.966338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:02.966545 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:02.966458    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:03.966663 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:03.966627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:03.967137 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:03.966767    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:03.967137 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:03.966812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:03.967137 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:03.966898    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:04.966213 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:04.966184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:04.966484 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:04.966273    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:05.499330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:05.499292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:05.499330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:05.499325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:05.499350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499472    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499543    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret podName:d128c5e4-fb0c-486a-a7ef-fa8545746825 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:37.499528475 +0000 UTC m=+66.258731542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret") pod "global-pull-secret-syncer-fpnrk" (UID: "d128c5e4-fb0c-486a-a7ef-fa8545746825") : object "kube-system"/"original-pull-secret" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499473    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499606    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs podName:bcfd05a2-2670-445f-8d81-412fa24a1d60 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:37.499595076 +0000 UTC m=+66.258798147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs") pod "network-metrics-daemon-lz9rf" (UID: "bcfd05a2-2670-445f-8d81-412fa24a1d60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499479    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499632    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499647    2574 projected.go:196] Error preparing data for projected volume kube-api-access-wjsv5 for pod openshift-network-diagnostics/network-check-target-zr4h4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:29:05.499834 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.499685    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5 podName:6b40159c-efef-4f32-91d3-7f0d6a74a146 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:37.499674778 +0000 UTC m=+66.258877848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjsv5" (UniqueName: "kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5") pod "network-check-target-zr4h4" (UID: "6b40159c-efef-4f32-91d3-7f0d6a74a146") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 23:29:05.966143 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:05.966100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:05.966350 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.966226    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:05.966350 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:05.966256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:05.966350 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:05.966321    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:06.966361 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:06.966325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:06.966800 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:06.966455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:07.966223 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:07.966186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:07.966437 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:07.966297    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:07.966437 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:07.966347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:07.966798 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:07.966459    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:08.966372 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:08.966336    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:08.966545 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:08.966455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:09.966356 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:09.966326    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:09.966576 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:09.966462    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:09.966576 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:09.966492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:09.966960 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:09.966593    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:10.966051 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:10.966024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:10.966224 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:10.966119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:11.966422 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:11.966379    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:11.966828 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:11.966472    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:11.966828 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:11.966526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:11.966828 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:11.966615    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:12.966534 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:12.966505    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:12.966903 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:12.966614    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:13.483322 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.483280    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr4h4"]
Nov 03 23:29:13.483504 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.483438    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:13.483573 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:13.483548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:13.486793 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.486767    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fpnrk"]
Nov 03 23:29:13.486932 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.486858    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:13.486994 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:13.486935    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:13.487646 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.487616    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lz9rf"]
Nov 03 23:29:13.487758 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.487741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:13.487875 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:13.487850    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:13.923665 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:13.923638    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-784mp_04000032-9127-46a5-ba82-dcbb72b1e394/dns-node-resolver/0.log"
Nov 03 23:29:14.966335 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:14.966300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:14.966744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:14.966341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:14.966744 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:14.966345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:14.966744 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:14.966445    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:14.966744 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:14.966505    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:14.966744 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:14.966570    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:15.127138 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:15.127115    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6rhdg_a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/node-ca/0.log"
Nov 03 23:29:16.965995 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:16.965816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:16.966451 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:16.965835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:16.966451 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:16.966072    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lz9rf" podUID="bcfd05a2-2670-445f-8d81-412fa24a1d60"
Nov 03 23:29:16.966451 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:16.965881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:16.966451 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:16.966142    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr4h4" podUID="6b40159c-efef-4f32-91d3-7f0d6a74a146"
Nov 03 23:29:16.966451 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:16.966211    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fpnrk" podUID="d128c5e4-fb0c-486a-a7ef-fa8545746825"
Nov 03 23:29:17.553184 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.553150    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeReady"
Nov 03 23:29:17.553331 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.553300    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 23:29:17.618818 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.618727    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ngzx9"]
Nov 03 23:29:17.620520 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.620502    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cswbt"]
Nov 03 23:29:17.620646 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.620627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.622272 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.622255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.634175 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.634155    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 23:29:17.637665 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.637642    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 23:29:17.637761 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.637715    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 23:29:17.637761 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.637721    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 23:29:17.637834 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.637766    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 23:29:17.644775 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.644749    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ngzx9"]
Nov 03 23:29:17.645229 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.645206    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cswbt"]
Nov 03 23:29:17.668016 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.667988    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4wqzj"]
Nov 03 23:29:17.669708 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.669691    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.684400 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.684364    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wqzj"]
Nov 03 23:29:17.726271 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.726234    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\""
Nov 03 23:29:17.726463 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.726310    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 23:29:17.726463 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.726316    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\""
Nov 03 23:29:17.726463 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.726400    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\""
Nov 03 23:29:17.732481 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.730025    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 23:29:17.738252 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.738234    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 23:29:17.738315 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.738253    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 23:29:17.786601 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.786674 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.786674 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.786674 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.786792 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm26\" (UniqueName: \"kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.786888 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcjt\" (UniqueName: \"kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.786923 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6k2\" (UniqueName: \"kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.786962 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.786962 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.787025 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.786997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.787071 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.787060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.887813 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.887813 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.887813 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888024 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.888024 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.888024 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.887964    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888024 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm26\" (UniqueName: \"kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.888169 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcjt\" (UniqueName: \"kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888169 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6k2\" (UniqueName: \"kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.888169 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888169 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888158    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888352 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888352 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.888352 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888323    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.888765 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888740    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.888897 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.888855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.891634 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.891614    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.891734 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.891617    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.891734 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.891706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.900158 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.900131    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6k2\" (UniqueName: \"kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.901896 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.901876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcjt\" (UniqueName: \"kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.908199 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.908172    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm26\" (UniqueName: \"kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:17.933442 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.933415    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:17.939015 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.938981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:17.979298 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:17.979094    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:18.268972 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.268948    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ngzx9"]
Nov 03 23:29:18.271327 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.271304    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cswbt"]
Nov 03 23:29:18.272368 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:18.272342    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f71c198_ea98_4678_8297_3d93e18fc042.slice/crio-dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d WatchSource:0}: Error finding container dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d: Status 404 returned error can't find the container with id dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d
Nov 03 23:29:18.276966 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:18.276947    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda378b9cf_dc33_42fa_8c6e_de8497ba0c2f.slice/crio-e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5 WatchSource:0}: Error finding container e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5: Status 404 returned error can't find the container with id e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5
Nov 03 23:29:18.342835 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.342815    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wqzj"]
Nov 03 23:29:18.354512 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:18.354484    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ed813d_7b33_4539_8d6f_657a80430229.slice/crio-1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef WatchSource:0}: Error finding container 1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef: Status 404 returned error can't find the container with id 1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef
Nov 03 23:29:18.966628 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.966590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:18.966828 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.966652    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:18.967423 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.967192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:18.972752 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.972733    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\""
Nov 03 23:29:18.973002 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.972985    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 23:29:18.973150 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.973133    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 23:29:18.973216 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.973192    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\""
Nov 03 23:29:18.973272 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.973216    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 23:29:18.973312 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.973271    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\""
Nov 03 23:29:18.973353 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:18.973311    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 23:29:19.178005 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:19.177963    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngzx9" event={"ID":"0f71c198-ea98-4678-8297-3d93e18fc042","Type":"ContainerStarted","Data":"dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d"}
Nov 03 23:29:19.179617 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:19.179589    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wqzj" event={"ID":"b9ed813d-7b33-4539-8d6f-657a80430229","Type":"ContainerStarted","Data":"1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef"}
Nov 03 23:29:19.181224 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:19.181204    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cswbt" event={"ID":"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f","Type":"ContainerStarted","Data":"4cae7ccd6d8daea6dcaff49c50e3f78ba0cacbe557f32212367f6c4e0656d684"}
Nov 03 23:29:19.181329 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:19.181256    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cswbt" event={"ID":"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f","Type":"ContainerStarted","Data":"df31d67aa7bbdf2e28435eab7e6d9edb8791174b6ca896556ebb0dce8fdeec6b"}
Nov 03 23:29:19.181329 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:19.181286    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cswbt" event={"ID":"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f","Type":"ContainerStarted","Data":"e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5"}
Nov 03 23:29:21.194010 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.193969    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cswbt" event={"ID":"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f","Type":"ContainerStarted","Data":"d62a2d18d7c3bd06b88c993b13ec6c3d38e09ef9fb7d78c957fd62da4b59a0cb"}
Nov 03 23:29:21.195740 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.195716    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngzx9" event={"ID":"0f71c198-ea98-4678-8297-3d93e18fc042","Type":"ContainerStarted","Data":"9cccd3196ee428ed933b8d0ca44b69af7913f7b104e72f1e0f06e821be42e1aa"}
Nov 03 23:29:21.195848 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.195760    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ngzx9" event={"ID":"0f71c198-ea98-4678-8297-3d93e18fc042","Type":"ContainerStarted","Data":"f01652933cb07267abdf01d05288416db551e945a47df6dd4dc74f6e7f3df496"}
Nov 03 23:29:21.195848 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.195794    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:21.213159 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.213120    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cswbt" podStartSLOduration=1.936134647 podStartE2EDuration="4.21310362s" totalImagesPullingTime="2.276968973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:29:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:21.212531684 +0000 UTC m=+49.971734755" watchObservedRunningTime="2026-11-03 23:29:21.21310362 +0000 UTC m=+49.972306689"
Nov 03 23:29:21.231676 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.231632    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-ngzx9" podStartSLOduration=2.224972664 podStartE2EDuration="4.231617667s" totalImagesPullingTime="2.006645003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:21.230672421 +0000 UTC m=+49.989875490" watchObservedRunningTime="2026-11-03 23:29:21.231617667 +0000 UTC m=+49.990820736"
Nov 03 23:29:21.898049 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.898017    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"]
Nov 03 23:29:21.899919 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.899904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:21.908143 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.908122    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"]
Nov 03 23:29:21.909953 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.909939    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917293    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917301    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8sbct\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917301    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917371    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917436    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917695    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917783    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xtxcc\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.917810    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 23:29:21.918652 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.918559    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"]
Nov 03 23:29:21.919976 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.919956    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6cdd9"]
Nov 03 23:29:21.920925 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.920899    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 23:29:21.922696 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.922678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:21.928037 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.928016    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"]
Nov 03 23:29:21.951229 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.951206    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 23:29:21.951229 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.951220    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fw76h\""
Nov 03 23:29:21.954674 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.951804    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 23:29:21.990090 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.990061    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 23:29:21.990239 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:21.990184    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 23:29:22.021330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.021330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzrj\" (UniqueName: \"kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.021559 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.021559 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.021559 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.021559 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.021694 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021561    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021694 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021694 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021795 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5h2p\" (UniqueName: \"kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.021795 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021795 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021795 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021927 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.021927 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.021927 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd87g\" (UniqueName: \"kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.022028 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.022028 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.022028 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.021993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123155 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzrj\" (UniqueName: \"kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.123359 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5h2p\" (UniqueName: \"kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.123690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.123565    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124150 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124117    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124236 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124117    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.124236 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124236 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124127    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124419 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124254    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124419 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.124419 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124419 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd87g\" (UniqueName: \"kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124419 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124370    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124655 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.124655 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.124655 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.124655 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124583    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.124924 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.124904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.125246 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.125223    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.125323 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.125259    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.128280 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.128413 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128288    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.128413 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.128413 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128366    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.128550 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.128695 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.128677    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.134861 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.134837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5h2p\" (UniqueName: \"kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.140797 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.140771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd87g\" (UniqueName: \"kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.152379 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.152324    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzrj\" (UniqueName: \"kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.200049 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.200005    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wqzj" event={"ID":"b9ed813d-7b33-4539-8d6f-657a80430229","Type":"ContainerStarted","Data":"a37adcc7bcd3b3317e2fc0b72309e0e40892a5420533f7803d257f9f685952e8"}
Nov 03 23:29:22.212051 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.212016    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:22.225673 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.225651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:22.227098 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.227054    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4wqzj" podStartSLOduration=2.356857348 podStartE2EDuration="5.227039683s" totalImagesPullingTime="2.870182335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:22.225584324 +0000 UTC m=+50.984787389" watchObservedRunningTime="2026-11-03 23:29:22.227039683 +0000 UTC m=+50.986242750"
Nov 03 23:29:22.241458 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.238796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:22.250495 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:22.250455    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026dcdf5_84bf_47d3_a512_6eef0ae86f5c.slice/crio-9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838 WatchSource:0}: Error finding container 9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838: Status 404 returned error can't find the container with id 9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838
Nov 03 23:29:22.490579 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.490493    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"]
Nov 03 23:29:22.494111 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:22.494084    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a9a502_a9d0_448d_ae12_e7fa44dc3c2f.slice/crio-bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63 WatchSource:0}: Error finding container bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63: Status 404 returned error can't find the container with id bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63
Nov 03 23:29:22.521102 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.519495    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"]
Nov 03 23:29:22.522125 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:22.522089    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf178f2de_cbfa_41db_9568_c52a97e0ec2d.slice/crio-c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c WatchSource:0}: Error finding container c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c: Status 404 returned error can't find the container with id c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c
Nov 03 23:29:22.901721 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.901676    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:29:22.904690 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.904664    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:22.911086 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911060    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 23:29:22.911318 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911302    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 23:29:22.911568 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911549    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 23:29:22.911685 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911613    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 23:29:22.911750 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911718    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 23:29:22.911826 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911812    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 23:29:22.911903 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.911891    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 23:29:22.912280 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.912044    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 23:29:22.912280 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.912158    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2dmvx\""
Nov 03 23:29:22.912280 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.912233    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 23:29:22.947819 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:22.947779    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:29:23.031921 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.031882    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032104 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.031958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032104 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032104 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032104 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032277 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032277 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032277 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032414 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032452 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032494 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032537 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.032570 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.032537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133556 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133699 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133699 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133699 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.133872 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.134290 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.134290 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.133925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.136375 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.136263    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.136635 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.136614    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.136751 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:23.136733    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle podName:7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:23.636714766 +0000 UTC m=+52.395917834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90") : configmap references non-existent config key: ca-bundle.crt
Nov 03 23:29:23.137586 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.137562    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.138625 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.138601    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.138944 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.138923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.141125 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.141100    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.141662 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.141633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.148854 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.148803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.148992 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.148957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.148992 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.148984    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.149379 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.149359    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.151240 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.151216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.206733 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.206639    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdd9" event={"ID":"026dcdf5-84bf-47d3-a512-6eef0ae86f5c","Type":"ContainerStarted","Data":"9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838"}
Nov 03 23:29:23.208671 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.208637    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb" event={"ID":"f178f2de-cbfa-41db-9568-c52a97e0ec2d","Type":"ContainerStarted","Data":"c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c"}
Nov 03 23:29:23.211812 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.211785    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj" event={"ID":"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f","Type":"ContainerStarted","Data":"667d166edf385db47481cd96693e283b1f03a531966458bbf5ea28498c67a812"}
Nov 03 23:29:23.211926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.211840    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj" event={"ID":"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f","Type":"ContainerStarted","Data":"1ed2490eeaee85eaf14eba4b625629a1b7481858f1aadd09b4f7c9d854592b03"}
Nov 03 23:29:23.211926 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.211872    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj" event={"ID":"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f","Type":"ContainerStarted","Data":"bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63"}
Nov 03 23:29:23.638647 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.638595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:23.638840 ip-10-0-143-134 kubenswrapper[2574]: E1103 23:29:23.638758    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle podName:7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:24.638738146 +0000 UTC m=+53.397941217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90") : configmap references non-existent config key: ca-bundle.crt
Nov 03 23:29:23.933102 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.933066    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8b55658f-66dwg"]
Nov 03 23:29:23.937611 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.937529    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:23.944682 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.944658    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e0s0ptbrqipi7\""
Nov 03 23:29:23.948697 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.948672    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 23:29:23.948786 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.948721    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 23:29:23.948858 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.948844    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mlhdz\""
Nov 03 23:29:23.948957 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.948945    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 23:29:23.949038 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.949027    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 23:29:23.949081 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.949071    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 23:29:23.961237 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:23.959346    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8b55658f-66dwg"]
Nov 03 23:29:24.042600 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042600 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042836 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042836 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042836 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042939 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042980 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.042980 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xgr\" (UniqueName: \"kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.043046 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.042982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143561 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143561 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143772 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143831 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143882 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143945 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.143998 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.143977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.144048 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.144002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xgr\" (UniqueName: \"kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.144101 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.144055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.144469 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.144231    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.144955 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.144929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.146628 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.146605    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.146714 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.146637    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.146787 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.146725    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.146893 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.146849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.147145 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.147125    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.147502 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.147486    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.155772 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.155742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xgr\" (UniqueName: \"kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.216503 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.216463    2574 generic.go:356] "Generic (PLEG): container finished" podID="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" containerID="fdd5a8a3274fef1b8d2ac18595d11c91fe40ca5a47421c07bba0e253b2640f07" exitCode=0
Nov 03 23:29:24.216961 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.216537    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdd9" event={"ID":"026dcdf5-84bf-47d3-a512-6eef0ae86f5c","Type":"ContainerDied","Data":"fdd5a8a3274fef1b8d2ac18595d11c91fe40ca5a47421c07bba0e253b2640f07"}
Nov 03 23:29:24.218903 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.218875    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb" event={"ID":"f178f2de-cbfa-41db-9568-c52a97e0ec2d","Type":"ContainerStarted","Data":"fc7f2510c69536818df253080332b9c478744a19f13dad12148b65a15c63b50c"}
Nov 03 23:29:24.219021 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.218941    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb" event={"ID":"f178f2de-cbfa-41db-9568-c52a97e0ec2d","Type":"ContainerStarted","Data":"890e1fde8f0f5beb7a4b40e411f0fe9f8fc1a74d8bc435891cbb8aa6d4041b01"}
Nov 03 23:29:24.219021 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.218972    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb" event={"ID":"f178f2de-cbfa-41db-9568-c52a97e0ec2d","Type":"ContainerStarted","Data":"ece8644d6b960419257e669d900311ff545f2cc08c505ad0cd866cc06d5895bf"}
Nov 03 23:29:24.221441 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.221420    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj" event={"ID":"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f","Type":"ContainerStarted","Data":"7cf22c233d655cac28979f7cf95ef06df37e0075dbc55e7978893297176cdfd4"}
Nov 03 23:29:24.255761 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.255362    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj" podStartSLOduration=2.184945323 podStartE2EDuration="3.255343072s" totalImagesPullingTime="1.070397749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:24.254979026 +0000 UTC m=+53.014182100" watchObservedRunningTime="2026-11-03 23:29:24.255343072 +0000 UTC m=+53.014546141"
Nov 03 23:29:24.266740 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.266718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:24.278030 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.277980    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb" podStartSLOduration=2.007382675 podStartE2EDuration="3.277959771s" totalImagesPullingTime="1.270577096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:24.277944272 +0000 UTC m=+53.037147346" watchObservedRunningTime="2026-11-03 23:29:24.277959771 +0000 UTC m=+53.037162862"
Nov 03 23:29:24.553311 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.553279    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8b55658f-66dwg"]
Nov 03 23:29:24.557884 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:24.557855    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfc458d_4b78_4b41_8ea4_79514fa4fe0f.slice/crio-24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a WatchSource:0}: Error finding container 24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a: Status 404 returned error can't find the container with id 24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a
Nov 03 23:29:24.647758 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.647724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:24.648464 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.648445    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:24.724758 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:24.724724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:25.010381 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.010299    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:29:25.014088 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:25.014063    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9d3aa6_8cfa_46fd_933e_28afd3cc3f90.slice/crio-213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b WatchSource:0}: Error finding container 213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b: Status 404 returned error can't find the container with id 213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b
Nov 03 23:29:25.234330 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.234283    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a"}
Nov 03 23:29:25.235708 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.235674    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b"}
Nov 03 23:29:25.238155 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.238129    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdd9" event={"ID":"026dcdf5-84bf-47d3-a512-6eef0ae86f5c","Type":"ContainerStarted","Data":"fc07ad26ec61a86337fe2425cd51b974790c68540b67db5ddf270074b6147791"}
Nov 03 23:29:25.238326 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.238175    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cdd9" event={"ID":"026dcdf5-84bf-47d3-a512-6eef0ae86f5c","Type":"ContainerStarted","Data":"1f02697529583fdaca5b30ba006d6ebf1265b8db1486c164b71daf44c3a4b57b"}
Nov 03 23:29:25.259542 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:25.259483    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6cdd9" podStartSLOduration=3.085928995 podStartE2EDuration="4.259466713s" totalImagesPullingTime="1.097683202s" totalInitContainerRuntime="75.854516ms" isStatefulPod=true podCreationTimestamp="2026-11-03 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:25.25841678 +0000 UTC m=+54.017619853" watchObservedRunningTime="2026-11-03 23:29:25.259466713 +0000 UTC m=+54.018669788"
Nov 03 23:29:26.244146 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.244059    2574 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab" exitCode=0
Nov 03 23:29:26.244625 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.244150    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"}
Nov 03 23:29:26.502632 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.502551    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"]
Nov 03 23:29:26.504856 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.504834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.520054 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.520027    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 23:29:26.520184 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.520094    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 23:29:26.520354 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.520333    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxkfg\""
Nov 03 23:29:26.520462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.520334    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 23:29:26.520462 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.520445    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 23:29:26.521785 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.521768    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"]
Nov 03 23:29:26.527634 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.527612    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3lqhglk32h6u6\""
Nov 03 23:29:26.567338 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567294    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbh7\" (UniqueName: \"kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567498 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567568 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567568 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567695 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567745 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.567816 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.567796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.586106 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.586072    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"]
Nov 03 23:29:26.588265 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.588240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.601585 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.601559    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"]
Nov 03 23:29:26.624829 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.624788    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2cj7f\""
Nov 03 23:29:26.624987 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.624853    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 23:29:26.668193 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668373 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668373 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.668373 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668373 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668583 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbh7\" (UniqueName: \"kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668583 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668688 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.668688 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.668636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2gj\" (UniqueName: \"kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.669025 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.669000    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.669241 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.669220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.669343 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.669322    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.670883 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.670861    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.671000 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.670975    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.671000 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.670984    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.679982 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.679954    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbh7\" (UniqueName: \"kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.769273 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.769246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2gj\" (UniqueName: \"kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.769352 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.769297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.779417 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.779371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.783294 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.783243    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2gj\" (UniqueName: \"kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:26.816332 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.816135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:26.904803 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:26.904561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:27.114207 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.114148    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"]
Nov 03 23:29:27.119194 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:27.119161    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89775551_7997_496e_84ac_55070f5b6665.slice/crio-fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6 WatchSource:0}: Error finding container fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6: Status 404 returned error can't find the container with id fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6
Nov 03 23:29:27.216627 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.216591    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"]
Nov 03 23:29:27.220044 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:27.220014    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5326b538_687b_4b39_b610_cf9b67362c02.slice/crio-c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea WatchSource:0}: Error finding container c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea: Status 404 returned error can't find the container with id c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea
Nov 03 23:29:27.249276 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.249240    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"e73839e74588f8c1842e3b0a70e9eca51fc7626744804af93b0e31dc61e3b7bb"}
Nov 03 23:29:27.249669 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.249314    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"b5e89adebcaa8d06ee17eebbe346a6f759f2b6c7d7dd242707a535c39f813cae"}
Nov 03 23:29:27.249669 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.249354    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"cbc40d802eac57ef13044840a91e56235c88a93214b8b6cdf7b4d8194005e5b6"}
Nov 03 23:29:27.250430 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.250404    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm" event={"ID":"5326b538-687b-4b39-b610-cf9b67362c02","Type":"ContainerStarted","Data":"c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea"}
Nov 03 23:29:27.251549 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:27.251528    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8" event={"ID":"89775551-7997-496e-84ac-55070f5b6665","Type":"ContainerStarted","Data":"fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6"}
Nov 03 23:29:28.260221 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:28.260171    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"7360cca2091a34709328d076cf0055bd81287771ea77208cae7fa3e0f2e88be8"}
Nov 03 23:29:28.260722 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:28.260262    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"dda50624410ec5578820f629216ca9125c214163d3e893139d0bd139111b2470"}
Nov 03 23:29:28.260722 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:28.260307    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" event={"ID":"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f","Type":"ContainerStarted","Data":"041a8f23371a3f360677bcca6027f17d36fe44b9bddaa7a6d2d9796c3b58eb45"}
Nov 03 23:29:28.261361 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:28.261338    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:28.289291 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:28.289232    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg" podStartSLOduration=2.370661889 podStartE2EDuration="5.289215157s" totalImagesPullingTime="2.918553268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:28.287850684 +0000 UTC m=+57.047053772" watchObservedRunningTime="2026-11-03 23:29:28.289215157 +0000 UTC m=+57.048418230"
Nov 03 23:29:29.270055 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270025    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"}
Nov 03 23:29:29.270424 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270104    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"}
Nov 03 23:29:29.270424 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270143    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"}
Nov 03 23:29:29.270424 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270183    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"}
Nov 03 23:29:29.270424 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270221    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"}
Nov 03 23:29:29.270424 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.270256    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerStarted","Data":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"}
Nov 03 23:29:29.272179 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.272148    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm" event={"ID":"5326b538-687b-4b39-b610-cf9b67362c02","Type":"ContainerStarted","Data":"bd6a83f1c67e5e2b8c83cf2774ca48d33aaf7a9be43c0ec1671f784473e1b241"}
Nov 03 23:29:29.272516 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.272498    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:29.277592 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.277573    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:29.308999 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.308942    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.231433616 podStartE2EDuration="7.308924434s" totalImagesPullingTime="3.067341601s" totalInitContainerRuntime="10.149217ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:29.307474288 +0000 UTC m=+58.066677362" watchObservedRunningTime="2026-11-03 23:29:29.308924434 +0000 UTC m=+58.068127508"
Nov 03 23:29:29.329251 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:29.325902    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm" podStartSLOduration=1.376751583 podStartE2EDuration="3.325881134s" totalImagesPullingTime="1.949129551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:29.323849585 +0000 UTC m=+58.083052659" watchObservedRunningTime="2026-11-03 23:29:29.325881134 +0000 UTC m=+58.085084207"
Nov 03 23:29:30.277037 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:30.276994    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8" event={"ID":"89775551-7997-496e-84ac-55070f5b6665","Type":"ContainerStarted","Data":"86625774a14862ab4fd395af90944369c7f0b417dde82765b15aa3cef53a2c27"}
Nov 03 23:29:31.149747 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:31.149716    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:31.185289 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:31.185223    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8" podStartSLOduration=3.037453756 podStartE2EDuration="5.185204088s" totalImagesPullingTime="2.147750332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:30.298979491 +0000 UTC m=+59.058182564" watchObservedRunningTime="2026-11-03 23:29:31.185204088 +0000 UTC m=+59.944407160"
Nov 03 23:29:31.203217 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:31.203187    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:35.283146 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:35.283117    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:37.579988 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.579951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:37.580487 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.580021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:37.580487 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.580145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:37.582146 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.582127    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 23:29:37.582667 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.582648    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 23:29:37.582741 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.582697    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 23:29:37.592907 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.592886    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 23:29:37.593094 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.593074    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:37.593182 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.593157    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:37.600108 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.600092    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\""
Nov 03 23:29:37.603402 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.603369    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:37.608158 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.608140    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:37.883282 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.883258    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lz9rf"]
Nov 03 23:29:37.885096 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:37.885064    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfd05a2_2670_445f_8d81_412fa24a1d60.slice/crio-96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f WatchSource:0}: Error finding container 96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f: Status 404 returned error can't find the container with id 96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f
Nov 03 23:29:37.885670 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.885648    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\""
Nov 03 23:29:37.891803 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.891786    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\""
Nov 03 23:29:37.893341 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.893325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:37.900521 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:37.900502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:38.166814 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:38.166737    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr4h4"]
Nov 03 23:29:38.170442 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:38.170374    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b40159c_efef_4f32_91d3_7f0d6a74a146.slice/crio-da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96 WatchSource:0}: Error finding container da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96: Status 404 returned error can't find the container with id da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96
Nov 03 23:29:38.198891 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:38.198865    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fpnrk"]
Nov 03 23:29:38.201770 ip-10-0-143-134 kubenswrapper[2574]: W1103 23:29:38.201738    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128c5e4_fb0c_486a_a7ef_fa8545746825.slice/crio-285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3 WatchSource:0}: Error finding container 285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3: Status 404 returned error can't find the container with id 285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3
Nov 03 23:29:38.309095 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:38.309058    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fpnrk" event={"ID":"d128c5e4-fb0c-486a-a7ef-fa8545746825","Type":"ContainerStarted","Data":"285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3"}
Nov 03 23:29:38.310214 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:38.310188    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr4h4" event={"ID":"6b40159c-efef-4f32-91d3-7f0d6a74a146","Type":"ContainerStarted","Data":"da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96"}
Nov 03 23:29:38.311236 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:38.311215    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz9rf" event={"ID":"bcfd05a2-2670-445f-8d81-412fa24a1d60","Type":"ContainerStarted","Data":"96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f"}
Nov 03 23:29:39.317010 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:39.316969    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz9rf" event={"ID":"bcfd05a2-2670-445f-8d81-412fa24a1d60","Type":"ContainerStarted","Data":"2266ab06efd76160e2b8ade89f2c46b7d62d2db9c7a36f14ac4aa0748d1900f5"}
Nov 03 23:29:39.317483 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:39.317039    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lz9rf" event={"ID":"bcfd05a2-2670-445f-8d81-412fa24a1d60","Type":"ContainerStarted","Data":"2c413adf72722c0b14a1ec3022dd2537a155a6964638384861e568652bcf0f57"}
Nov 03 23:29:39.348029 ip-10-0-143-134 kubenswrapper[2574]: I1103 23:29:39.347968    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lz9rf" podStartSLOduration=67.117530209 podStartE2EDuration="1m8.347950624s" totalImagesPullingTime="1.230420415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:39.34706661 +0000 UTC m=+68.106269682" watchObservedRunningTime="2026-11-03 23:29:39.347950624 +0000 UTC m=+68.107153696"
Nov 03 23:29:44.048790 ip-10-0-143-134 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 23:29:44.060864 ip-10-0-143-134 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 23:29:44.061054 ip-10-0-143-134 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 23:29:44.061255 ip-10-0-143-134 systemd[1]: kubelet.service: Consumed 4.280s CPU time.
Nov 03 23:29:44.076920 ip-10-0-143-134 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 23:29:44.138846 ip-10-0-143-134 kubenswrapper[6430]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:29:44.138846 ip-10-0-143-134 kubenswrapper[6430]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 23:29:44.138846 ip-10-0-143-134 kubenswrapper[6430]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:29:44.138846 ip-10-0-143-134 kubenswrapper[6430]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 23:29:44.140993 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.140912    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.140993 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.140992    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.140993 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.140996    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141000    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141002    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141006    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141008    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141011    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141013    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141016    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141019    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141022    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141024    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141027    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141030    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141032    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141035    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141037    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141040    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141042    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141045    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.141091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141048    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141051    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141054    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141056    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141059    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141063    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141065    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141068    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141071    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141074    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141077    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141079    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141082    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141084    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141086    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141089    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141092    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141101    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141106    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.141588 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141109    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141112    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141116    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141119    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141121    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141125    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141129    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141132    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141135    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141138    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141141    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141144    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141146    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141148    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141151    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141155    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141157    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141160    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141164    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.142058 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141166    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141169    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141171    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141174    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141176    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141179    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141181    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141184    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141186    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141189    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141191    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141194    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141196    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141199    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141201    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141204    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141206    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141209    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141211    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141214    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.142576 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141217    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141219    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141222    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141224    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141228    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141231    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141234    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141236    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141240    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141243    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141246    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141248    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141251    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141254    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141257    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141259    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141262    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141264    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141267    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141270    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.143066 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141273    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141275    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141278    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141281    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141283    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141286    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.141289    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142072    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142079    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142082    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142085    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142088    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142091    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142093    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142096    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142100    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142103    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142106    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142108    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.143592 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142111    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142113    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142116    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142118    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142121    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142123    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142127    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142129    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142132    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142135    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142137    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142140    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142143    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142145    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142148    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142150    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142153    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142156    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142158    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142161    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.144091 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142164    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142167    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142169    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142172    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142174    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142177    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142179    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142182    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142185    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142188    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142190    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142193    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142195    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142198    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142203    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142207    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142211    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142214    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142218    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.144610 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142229    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142232    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142235    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142237    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142240    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142242    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142245    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142247    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142250    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142252    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142255    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142257    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142260    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142264    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142266    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142269    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142271    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142274    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142277    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.145097 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142279    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142282    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142285    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142288    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142290    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142293    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142295    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142298    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142301    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142304    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142306    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142309    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142312    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142315    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142317    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142320    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142323    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142325    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142328    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142330    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.145632 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142333    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142335    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142338    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142341    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142343    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142346    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142348    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142351    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142354    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142356    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142359    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142361    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142365    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142367    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142370    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142372    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142877    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142882    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142885    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.146132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142888    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142890    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142893    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142895    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142898    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142901    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142903    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142906    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142909    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142912    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142914    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142920    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142922    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142925    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142928    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142930    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142933    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142936    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142938    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142941    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.146752 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142944    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142946    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142949    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142952    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142954    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142957    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142959    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142962    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142964    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142967    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142970    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142972    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142975    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142978    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142980    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142983    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142985    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142988    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142991    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142993    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.147316 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142996    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.142999    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143001    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143007    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143011    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143014    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143018    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143022    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143025    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143027    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143030    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143033    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143036    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143038    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143041    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143045    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143047    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143050    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.147902 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143053    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143055    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143058    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143060    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143063    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143065    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143068    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143071    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143074    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143077    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143079    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143082    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143085    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143088    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143090    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143093    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143096    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143100    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143103    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143106    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.148408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143108    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143111    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143113    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143116    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143119    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143121    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143124    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143126    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143129    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143132    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143134    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143137    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143139    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143153    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143155    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143158    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143161    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143163    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143166    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143169    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.148961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143171    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143174    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143176    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143179    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.143182    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143273    6430 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143280    6430 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143285    6430 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143291    6430 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143297    6430 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143301    6430 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143305    6430 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143309    6430 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143312    6430 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143316    6430 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143319    6430 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143322    6430 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143325    6430 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143328    6430 flags.go:64] FLAG: --cgroup-root=""
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143331    6430 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143334    6430 flags.go:64] FLAG: --client-ca-file=""
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143338    6430 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143348    6430 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143354    6430 flags.go:64] FLAG: --cluster-domain=""
Nov 03 23:29:44.149778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143357    6430 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143360    6430 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143363    6430 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143367    6430 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143371    6430 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143374    6430 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143377    6430 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143380    6430 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143398    6430 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143401    6430 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143404    6430 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143407    6430 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143410    6430 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143415    6430 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143418    6430 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143421    6430 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143424    6430 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143429    6430 flags.go:64] FLAG: --enable-server="true"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143432    6430 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143437    6430 flags.go:64] FLAG: --event-burst="100"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143440    6430 flags.go:64] FLAG: --event-qps="50"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143443    6430 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143446    6430 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143451    6430 flags.go:64] FLAG: --eviction-hard=""
Nov 03 23:29:44.150597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143455    6430 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143458    6430 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143461    6430 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143465    6430 flags.go:64] FLAG: --eviction-soft=""
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143469    6430 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143472    6430 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143475    6430 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143478    6430 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143481    6430 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143484    6430 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143487    6430 flags.go:64] FLAG: --feature-gates=""
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143491    6430 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143494    6430 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143497    6430 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143500    6430 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143503    6430 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143506    6430 flags.go:64] FLAG: --help="false"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143509    6430 flags.go:64] FLAG: --hostname-override="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143513    6430 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143516    6430 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143519    6430 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143522    6430 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143526    6430 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 23:29:44.151203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143529    6430 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143532    6430 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143536    6430 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143539    6430 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143542    6430 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143546    6430 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143549    6430 flags.go:64] FLAG: --kube-reserved=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143552    6430 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143557    6430 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143560    6430 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143563    6430 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143566    6430 flags.go:64] FLAG: --lock-file=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143568    6430 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143571    6430 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143575    6430 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143580    6430 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143583    6430 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143586    6430 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143589    6430 flags.go:64] FLAG: --logging-format="text"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143592    6430 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143596    6430 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143599    6430 flags.go:64] FLAG: --manifest-url=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143602    6430 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143606    6430 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143610    6430 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 23:29:44.151798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143614    6430 flags.go:64] FLAG: --max-pods="110"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143617    6430 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143621    6430 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143624    6430 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143627    6430 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143630    6430 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143633    6430 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143635    6430 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143642    6430 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143647    6430 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143650    6430 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143659    6430 flags.go:64] FLAG: --pod-cidr=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143663    6430 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143665    6430 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143669    6430 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143673    6430 flags.go:64] FLAG: --port="10250"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143676    6430 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143679    6430 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04a95d504492b9e3a"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143682    6430 flags.go:64] FLAG: --qos-reserved=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143686    6430 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143689    6430 flags.go:64] FLAG: --register-node="true"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143692    6430 flags.go:64] FLAG: --register-with-taints=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143695    6430 flags.go:64] FLAG: --registry-burst="10"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143698    6430 flags.go:64] FLAG: --registry-qps="5"
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143702    6430 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143704    6430 flags.go:64] FLAG: --reserved-memory=""
Nov 03 23:29:44.152438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143708    6430 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143712    6430 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143715    6430 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143717    6430 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143720    6430 flags.go:64] FLAG: --runonce="false"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143723    6430 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143726    6430 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143729    6430 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143732    6430 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143735    6430 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143738    6430 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143742    6430 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143745    6430 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143748    6430 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143751    6430 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143756    6430 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143759    6430 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143761    6430 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143770    6430 flags.go:64] FLAG: --system-cgroups=""
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143773    6430 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143779    6430 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143783    6430 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143786    6430 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143800    6430 flags.go:64] FLAG: --tls-min-version=""
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143803    6430 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 23:29:44.153185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143807    6430 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143810    6430 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143813    6430 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143816    6430 flags.go:64] FLAG: --v="2"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143824    6430 flags.go:64] FLAG: --version="false"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143829    6430 flags.go:64] FLAG: --vmodule=""
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143832    6430 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.143836    6430 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144017    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144021    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144024    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144027    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144030    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144033    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144035    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144038    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144041    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144044    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144047    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144049    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144052    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.153929 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144055    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144059    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144062    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144065    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144067    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144071    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144074    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144078    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144081    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144084    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144086    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144089    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144092    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144095    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144098    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144100    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144103    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144106    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144108    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144111    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.154481 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144114    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144116    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144119    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144122    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144124    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144127    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144130    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144132    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144135    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144137    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144140    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144142    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144145    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144148    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144151    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144154    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144157    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144159    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144163    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144166    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.154974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144169    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144172    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144174    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144177    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144180    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144182    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144185    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144187    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144190    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144193    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144195    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144198    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144200    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144203    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144205    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144208    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144210    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144213    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144215    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.155738 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144218    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144221    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144223    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144226    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144229    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144231    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144236    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144239    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144242    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144244    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144247    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144249    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144255    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144257    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144260    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144264    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144268    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144272    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144276    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144279    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.156274 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144282    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144285    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144287    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144290    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144293    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144295    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144298    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144300    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144303    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144305    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144308    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144310    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144313    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144316    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.144342    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144375    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.156791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144379    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144382    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144397    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144401    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144404    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144407    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144410    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144413    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144416    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144423    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144426    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144428    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144431    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144434    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144436    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144439    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144442    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144444    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144448    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.157188 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144452    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144468    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144472    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144476    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144479    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144482    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144485    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144489    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144492    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144495    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144498    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144501    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144503    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144506    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144509    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144511    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144515    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144517    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144520    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.157881 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144523    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144525    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144529    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144532    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144535    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144538    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144540    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144543    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144546    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144549    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144553    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144558    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144563    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144566    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144570    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144575    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144579    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144584    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144587    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.158467 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144589    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144592    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144594    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144597    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144600    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144602    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144605    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144607    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144610    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144613    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144617    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144620    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144623    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144626    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144628    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144633    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144636    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144638    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144647    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144650    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.158988 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144653    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144655    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144658    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144661    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144663    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144666    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144668    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144671    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144674    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144676    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144681    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144683    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144686    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144689    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144691    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144694    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144696    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144699    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144701    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.159572 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144704    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144706    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144709    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144713    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144716    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144718    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144721    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144723    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.144727    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.149373    6430 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.149410    6430 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.149435    6430 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.149445    6430 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149655    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149664    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149670    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149675    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149686    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149690    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149694    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.160045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149699    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149703    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149708    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149712    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149718    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149722    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149727    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149732    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149737    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149742    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.149754    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150289    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150298    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150304    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150308    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150311    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150314    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150317    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150320    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150324    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.160555 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150327    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150330    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150332    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150335    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150338    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150340    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150343    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150346    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150349    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150352    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150355    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150359    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150361    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150364    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150367    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150370    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150373    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150376    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150378    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150381    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.161049 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150384    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150403    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150408    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150411    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150414    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150417    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150419    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150422    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150426    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150428    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150431    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150434    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150438    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150443    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150446    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150449    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150452    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150455    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150458    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.161613 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150461    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150465    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150468    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150471    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150474    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150477    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150480    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150482    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150485    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150488    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150490    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150493    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150496    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150499    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150502    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150504    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150507    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150509    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150512    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.162221 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150516    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150520    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150523    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150527    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150530    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150533    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150536    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150539    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150541    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150544    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150547    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150550    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150552    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150556    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150559    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150561    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150564    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150567    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150569    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150572    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.162879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150574    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.150609    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150772    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150778    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150782    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150784    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150787    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150796    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150799    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150802    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150805    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150807    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150810    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150813    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150816    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150819    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.163679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150821    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150825    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150827    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150830    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150832    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150835    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150837    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150840    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150843    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150846    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150848    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150850    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150853    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150855    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150858    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150861    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150864    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150866    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150869    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.164177 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150872    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150874    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150877    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150879    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150882    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150886    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150888    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150891    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150895    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150899    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150902    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150905    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150907    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150910    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150913    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150915    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150918    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150921    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150924    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150927    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.164764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150929    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150932    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150934    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150937    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150939    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150942    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150944    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150947    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150949    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150952    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150954    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150957    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150960    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150964    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150967    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150969    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150972    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150974    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150978    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.165453 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150980    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150983    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150985    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150988    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150991    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150993    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150996    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.150999    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151002    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151004    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151007    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151010    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151012    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151015    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151018    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151020    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151023    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151026    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151028    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151031    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.166076 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151033    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151036    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151038    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151041    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151043    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151046    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151048    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151051    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151054    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151057    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151059    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151062    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151065    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151068    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.151090    6430 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 23:29:44.166796 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151121    6430 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151124    6430 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151127    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151130    6430 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151133    6430 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151136    6430 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151139    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151141    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151144    6430 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151147    6430 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151149    6430 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151152    6430 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151155    6430 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151157    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151160    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151162    6430 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151165    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151167    6430 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151170    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 23:29:44.167408 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151172    6430 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151175    6430 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151178    6430 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151180    6430 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151184    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151186    6430 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151189    6430 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151192    6430 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151194    6430 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151197    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151199    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151203    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151205    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151208    6430 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151211    6430 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151214    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151216    6430 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151219    6430 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151222    6430 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 23:29:44.167984 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151224    6430 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151227    6430 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151229    6430 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151232    6430 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151235    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151237    6430 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151239    6430 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151242    6430 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151244    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151247    6430 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151249    6430 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151252    6430 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151255    6430 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151257    6430 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151260    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151263    6430 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151267    6430 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151272    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151275    6430 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151278    6430 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 23:29:44.168474 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151281    6430 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151283    6430 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151286    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151288    6430 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151291    6430 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151295    6430 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151298    6430 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151301    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151303    6430 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151306    6430 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151308    6430 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151311    6430 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151313    6430 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151316    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151318    6430 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151321    6430 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151323    6430 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151326    6430 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151328    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 23:29:44.168974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151331    6430 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151335    6430 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151338    6430 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151351    6430 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151355    6430 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151358    6430 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151360    6430 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151363    6430 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151366    6430 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151369    6430 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151377    6430 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151379    6430 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151382    6430 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151398    6430 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151403    6430 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151407    6430 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151412    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151415    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151418    6430 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 23:29:44.169611 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151421    6430 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151424    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151427    6430 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151429    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151432    6430 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151436    6430 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151438    6430 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151441    6430 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151443    6430 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:29:44.151446    6430 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.151498    6430 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.152512    6430 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.152563    6430 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.152758    6430 server.go:1014] "Starting client certificate rotation"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.152899    6430 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.152952    6430 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:31 +0000 UTC" deadline="2028-04-12 22:26:49.475888155 +0000 UTC"
Nov 03 23:29:44.170100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.153000    6430 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12622h57m5.322891567s"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.153289    6430 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.154302    6430 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.155670    6430 log.go:25] "Validated CRI v1 runtime API"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.157623    6430 log.go:25] "Validated CRI v1 image API"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.158397    6430 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 23:29:44.170515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.160639    6430 fs.go:135] Filesystem UUIDs: map[2551f046-6da1-4d05-b790-453b79579363:/dev/nvme0n1p3 5cb435c4-7145-4bfb-b970-a05d5c3bc160:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 23:29:44.170700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.160658    6430 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3/userdata/shm major:0 minor:435 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96/userdata/shm major:0 minor:433 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~projected/kube-api-access-vd87g:{mountpoint:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~projected/kube-api-access-vd87g major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04000032-9127-46a5-ba82-dcbb72b1e394/volumes/kubernetes.io~projected/kube-api-access-4z9bw:{mountpoint:/var/lib/kubelet/pods/04000032-9127-46a5-ba82-dcbb72b1e394/volumes/kubernetes.io~projected/kube-api-access-4z9bw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/volumes/kubernetes.io~projected/kube-api-access-grwzf:{mountpoint:/var/lib/kubelet/pods/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/volumes/kubernetes.io~projected/kube-api-access-grwzf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e628017-63f5-4329-86b0-f07e84304f96/volumes/kubernetes.io~projected/kube-api-access-tgpzz:{mountpoint:/var/lib/kubelet/pods/0e628017-63f5-4329-86b0-f07e84304f96/volumes/kubernetes.io~projected/kube-api-access-tgpzz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~projected/kube-api-access-gg6k2:{mountpoint:/var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~projected/kube-api-access-gg6k2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~projected/kube-api-access-6bzrj:{mountpoint:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~projected/kube-api-access-6bzrj major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~projected/kube-api-access-6j2gj:{mountpoint:/var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~projected/kube-api-access-6j2gj major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~projected/kube-api-access-pk6pt:{mountpoint:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~projected/kube-api-access-pk6pt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5eeb5df1-2d70-4963-ab3f-503518320882/volumes/kubernetes.io~projected/kube-api-access-tkpsp:{mountpoint:/var/lib/kubelet/pods/5eeb5df1-2d70-4963-ab3f-503518320882/volumes/kubernetes.io~projected/kube-api-access-tkpsp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b40159c-efef-4f32-91d3-7f0d6a74a146/volumes/kubernetes.io~projected/kube-api-access-wjsv5:{mountpoint:/var/lib/kubelet/pods/6b40159c-efef-4f32-91d3-7f0d6a74a146/volumes/kubernetes.io~projected/kube-api-access-wjsv5 major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~empty-dir/config-out major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/kube-api-access-55bz4:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/kube-api-access-55bz4 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/config-volume major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~projected/kube-api-access-rwbh7:{mountpoint:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~projected/kube-api-access-rwbh7 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9570168c-b9aa-455f-981e-33d57c4b5671/volumes/kubernetes.io~projected/kube-api-access-t65tt:{mountpoint:/var/lib/kubelet/pods/9570168c-b9aa-455f-981e-33d57c4b5671/volumes/kubernetes.io~projected/kube-api-access-t65tt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/volumes/kubernetes.io~projected/kube-api-access-pg7v7:{mountpoint:/var/lib/kubelet/pods/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/volumes/kubernetes.io~projected/kube-api-access-pg7v7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~projected/kube-api-access-xdcjt:{mountpoint:/var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~projected/kube-api-access-xdcjt major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ddeb79-97e0-492a-a4d6-f50f96720fb7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b0ddeb79-97e0-492a-a4d6-f50f96720fb7/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~projected/kube-api-access-qfm26:{mountpoint:/var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~projected/kube-api-access-qfm26 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~projected/kube-api-access-v4rjq:{mountpoint:/var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~projected/kube-api-access-v4rjq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~secret/metrics-certs major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~projected/kube-api-access-h89x9:{mountpoint:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~projected/kube-api-access-h89x9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d128c5e4-fb0c-486a-a7ef-fa8545746825/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d128c5e4-fb0c-486a-a7ef-fa8545746825/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~projected/kube-api-access-t5xgr:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~projected/kube-api-access-t5xgr major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~projected/kube-api-access-k5h2p:{mountpoint:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~projected/kube-api-access-k5h2p major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3794e50bc4ed7b35756d068599664d1c5d8e67cb619c23dc9d733e511ee9c7fe/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bb2687c5e6892b80443714a145faf060d4f06aece406beee81800dd32010003d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1eff3dd339ff1c89f818fed192555196ce880d56f028893a39c6545342a7196b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0c9c9c1f2c332f642592f591ecce8fc9d114aa541d6a1bcb6fc96c79ff9ad3b2/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/00e9e39f6f5f924e6cb58d9c4ada3b1768ed442ac651a291809737ec4673415b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/709c4629df95d94c95ff58ae35433592edd869151f25bea83f7a097ebbed382b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/7dbd94bf3956ec08324fc643f40a2b0dd5835a9aa37cf696d01a27ae6354c4cf/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d1f685adfc2b638b1e24a5b1973ba3eeb7e6dcc9bd10792bd42fcd6d9f3a8a5b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ad108f3289977d8a28b6f66a9a7e81f77e521e94d3c39a22a0cd50ceaf41449f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7079f6bfb61a821b799533614e57868cdf5f20fd4fc4288fe8830ae0a33915a5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/ef55a9585916ce4b8a068a31c7daa183264f92cb8159a041b8c7c855c9f3601b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6c305e2e2013e3ed96d0bfd332c7d5d2a3271bac995911c2b5f9c5716cbbabba/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bc38fbd97ddd7876f9452b7309a91ed9b84c69ea0c338f9fe88a5983ed39a149/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d4fdda15f5780427a5fe4475797bfe9cd758edcb70eebc832bdfd7c5b5031645/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/88b93173a91ae0000ea8512ddffe72462cdb8056a23f7543e2bf6638a322e0c3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2c28212eee1e5761be71642a78fb6c0416770ba80467c39ae9bd822445925f8b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e934b3eafbbcdb63d2760fab9b3af18271a1fb06056ef3320489095ecf090741/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a0a61b8077b1d68f843d5520f360983826e6557c4b9f7aeb6c21c13912516653/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/be56d4561938b9f893b2edfd1d389f7f18a204cbc5336cb3d23459bf03246358/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d66562ce8810bafa710fe39c57cdef4c99c1d548abed429e74789ffef8f9347a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6ea356c0ab1222e98e0581d68651a6560676f89e0eeba276833b255b3f773de9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/9752f9e4c54a7e0da93cdf0d59391f3899f980c4a8b8bf26ba70d4e52db1d9e6/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c6ae57a6219d450d2985428cceb8f9e16654ae6402d546e33ed9e68d9c986e3f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/901ea866e4777c9d26983e339de4283829fde3b92c72adfa7a8a89af50503940/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/9d0579184dab0dabf3c3164774be96d6fcd58b149ed57a41ffd4a6e9a677e6e5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/a9d711ad0bc031a3c93747b4d1e6932d1ac9b189777ab84775d5c9d5f3039be5/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/6b50544a6ff9fe7c6770a97c32611fb5e49269c5e46f768953a6f450ede2c2e1/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f56bbe1ac2246054d05964624d7025be5beb730dcf3d2e8b7104d59b82f61509/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/8bb94801cafc89960c6258d8386d2a0631e4505107f15f1455cebecd780201e6/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/c15a33ea7cc3429f6cb8b00cc7ddc6aefc54f065829bd96c2f9286e56f0e873a/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/a5fd903f25b93a5b5a1477fcde94271e835b2d55f03331d5a5847d81d89b873c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/067f4684e69946347986c3954c83056ade101fd73f05941c9168fb814cf19221/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/7a503a6a820acd0a416a3176dbdd739f10ac9cd3518dbc733e4e7db1884f3739/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/eac9886adb782cba031ecae6adceaf731053e29b5f7409ac54ef7473304da45b/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/122943b28b93afc4f2a867a27dc4f12c8b6b2cb1512b53d4a0bb101f0be9f459/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/b5bf1d0217a8a16590b1f5e8c3d6072d944ed0402371204a89d397281bf8300e/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/4ad81f5b3f172444c6c04b6a335e4a97d0ebad7d4b2ce41be69c6adf5893a6ae/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/362cde91f16fb90486a1fa2d9fcf88f64c1bbed77fb3282997e38c18dedc62ff/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/8d99ded6cc5de8a853435b254abaa8c138bf9c7f5e00179a890b92409eb206c7/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/eeb6a7d296a590f76cd1d4786e3d413eef51764fe486830132af57a312317e4f/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/ce949a8e52f8fff1c24bdacc34929905e63d4916a45e195d399290bff0d60ebf/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/9a4aa0c674c28f56affcc6a1b9f1f2cb4d1857dd3b54c2a919805d1c38dc7f4f/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/6e521c7ccc4655310575b2513c6b27f84e99c4561fcd1188ca33a4f073c27357/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/3bfbd54cbfafefce1550e7cbb3a0d8ea46cfd2b09aa55b9430993a0b636efa16/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/fa7375e6e4c07424464953480e4b92dc2afe50feb7364a2dd24c5a700f50a935/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/41b23a171f41fe510a5b7c34243b12e4560e31e91021d19331a3444d7601776d/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/566886979d5b3eb65b2f170eb62d262097196f5ab08d08a982f873a9993829df/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/1a568c6bfa8581c1b9a49c9aa91d1d9f281708b781ef2cc0c77e03992ee49aa2/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/3752b52b6b89e53c85da7228a82561bd9bb4c76b0385eaf30235ea977d0d3365/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/9fe1e12c28c09d0a5deed495c5f69edabfd1599ce01280128e01eb5cc4f27ef3/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/c74d29ec250c1020d1398f977a06da96114d91fd06b7df38e9480d4de55e8dcb/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/787cbb033c5eafa35deef915816825436f36f5f5f760b9fa5f10b8ac8fb9ac99/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/6b71753a8018d566a287e4efe0f00643406028ec59ffe1bebbc6d41705deff08/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/33d3c30ae06b5a82dc7352da1e7f0e5444c552f1b2365a4721467f13abbe1825/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/dd84748d52992e4416811482d15ec846b03829a811859ba604e7454d04d45dc4/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/d575a4d7404a4ae0daa74d35aa9aac90a78e98c3edbe71e5812eab64f72525d5/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/47d78a23f1cfa3b3b1a0c1bb2e2defd6b6f8e2883e7b8c97dae4ac523c01cf93/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/93f0392c3c66050e98804cd390d858710d48776780dff18e6d2db6ef92a5f383/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/c5600dcc33b9041cd9d90b062552bb6fa1c610bcce55841e23c65c501458758a/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/d05e7ad51a622172c49eb8556ccfdb764824d257e2229bd799e3d43b0f5fc152/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/1a9bf928c9f901a249e9589a443c58c73918974c9735ae9ff47b5fb29fbe661f/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/2d4cb948a25c8fb5cde3a41faa8daa6c30698fc132060238a402698f3a6c81fe/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/324ff73f2215f442fdc1dd65188c1739811bfa4947eca0383d742747f3ad0690/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/80c0aad4b5e68166a4300b588916b990774996f89cb437df04d136ff4269bf45/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/997a5d69a86fa0e807828c9528c81ff2c35420161da45a4511a404c605bcd98e/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/b22d0d8ba3bd1f853ee454862cd66fa13342296a8c273ccd13a06f096c8f8ce2/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/351890f333789f5dbb25df268c64fc9a0c70494cbfa352e32522d7f35765eef3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/851286855bb8e0fbd64699207e563ba318f92faf68ce0e547363f5882e38dd13/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f61624f0f71a41c1881b6a2732feee04bcd6ca5dd76d16d27dc8f744ae2b8820/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6e43a5d14d82aeaa13d6722e93093fd3c2fb14ba2664a5841c3b571ffe3a4327/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/955a8775ffa44ca4230b1a25c024c2191e94bc8ed6388ec154a9e59ec61f3dc4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/1109fea3cded3f22512006e3cd586c179d21f49a8659e5ef66da4a4800e45115/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/73f63915e21ff7c72bb4b73e007b1ffce01da0a82469dbbcaafed4f26b5dfb24/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8503a4b7ef473258cf174f65b7c74fd4395b7526b8d533365716dea2f6a45f95/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d5184811e3e3dcbebf60befd7641f630b9579d9ddc5fb60e8b97a9f7578e6851/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 23:29:44.170954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.170351    6430 manager.go:217] Machine: {Timestamp:2026-11-03 23:29:44.169880194 +0000 UTC m=+0.060942122 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099612 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd3738a6e17481780ecdeb6b200df SystemUUID:ec2fd373-8a6e-1748-1780-ecdeb6b200df BootID:363ad058-b806-4a50-b5d1-117eb7f8a500 Filesystems:[{Device:/var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~projected/kube-api-access-qfm26 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf22b0e710824e2c225501bc96ecd63a8eac869fd840e7e8fee71cb7c5d051d/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09bdf8685b12873c341383c067a19bc925b5ede1e0ee30977efb42f11d37518c/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf9594d715907a34045d581dbe4d6a92976539315d58c3b068cefd5fbe0ace63/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24e06496d8ec7f2286df7e75d296530fe1ad840ef3e605194c41d0d61e1fd63a/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96c9c211a12896c2e1b28181e5dd78459387c0df4e87372af2f5722922af793f/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~projected/kube-api-access-h89x9 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~projected/kube-api-access-v4rjq DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ea0819b8526091a18ae6f3b8d3ee8fd97b715381bd7c836694f4245586231ef/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~projected/kube-api-access-6bzrj DeviceMajor:0 DeviceMinor:258 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4200e6eedf98a3063e72ef64fc63f508a2f641099d24d41e93755ccf871c46c/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5eeb5df1-2d70-4963-ab3f-503518320882/volumes/kubernetes.io~projected/kube-api-access-tkpsp DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90d433ceaf1d868093e8fa9d50094a1824484b47c8116a44cffa0bced60c9284/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~projected/kube-api-access-rwbh7 DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1cce19056447101004eb8f18b72b7dcd7e674191906f7a265d8618754fdeeea/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f71c198-ea98-4678-8297-3d93e18fc042/volumes/kubernetes.io~projected/kube-api-access-gg6k2 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~projected/kube-api-access-t5xgr DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6b40159c-efef-4f32-91d3-7f0d6a74a146/volumes/kubernetes.io~projected/kube-api-access-wjsv5 DeviceMajor:0 DeviceMinor:428 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d128c5e4-fb0c-486a-a7ef-fa8545746825/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:422 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da826a48d07d8ce651031a4525af169f16dce12ab80c9c5dd16c1fd9c8677f96/userdata/shm DeviceMajor:0 DeviceMinor:433 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5326b538-687b-4b39-b610-cf9b67362c02/volumes/kubernetes.io~projected/kube-api-access-6j2gj DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/kube-api-access-55bz4 DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d17f18175bf4fd84381a6504823b49c2a78180c0f7bfef919a767986cc0207c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/volumes/kubernetes.io~projected/kube-api-access-xdcjt DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bcfd05a2-2670-445f-8d81-412fa24a1d60/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:427 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a6c5a4e3cb27806608c1d5b67661909ecdb4a36bfa4b663d80d3987f3960966/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~projected/kube-api-access-vd87g DeviceMajor:0 DeviceMinor:257 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e628017-63f5-4329-86b0-f07e84304f96/volumes/kubernetes.io~projected/kube-api-access-tgpzz DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18055547ac299b9bd9cb1d33b35005a1ee51db1755307cb9b42304adc907a50d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/285a57dac700951fafba85595ad6260323dee9be97e9de64f10c0ab67e4651a3/userdata/shm DeviceMajor:0 DeviceMinor:435 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c17332ae-befe-4495-8904-af7ad229d464/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/026dcdf5-84bf-47d3-a512-6eef0ae86f5c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe3cd926e3b9ef153fe4a4e97211ceadcdb850cb532cf59785f0385e9a8916d6/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9570168c-b9aa-455f-981e-33d57c4b5671/volumes/kubernetes.io~projected/kube-api-access-t65tt DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/04000032-9127-46a5-ba82-dcbb72b1e394/volumes/kubernetes.io~projected/kube-api-access-4z9bw DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/89775551-7997-496e-84ac-55070f5b6665/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f178f2de-cbfa-41db-9568-c52a97e0ec2d/volumes/kubernetes.io~projected/kube-api-access-k5h2p DeviceMajor:0 DeviceMinor:256 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/volumes/kubernetes.io~projected/kube-api-access-grwzf DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70f6edca69a500b612496150e278434aa3c895a1be469897e3262963e608f719/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53f9d6a82a5b0c37c5235334443a028866ed8d42360b3dbdfc6e6985bd05d0d0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8fbc654ab5ad09550bb2bf37af57e6cfe66669dbee391ce261b547ff7ca6fc5/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ddeb79-97e0-492a-a4d6-f50f96720fb7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9ed813d-7b33-4539-8d6f-657a80430229/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/volumes/kubernetes.io~projected/kube-api-access-pg7v7 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9199731041e3ae75f50732b70daec4a5c5b4af0fdf1f5c2e2da88015bb88e838/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a32217f-0822-43ae-8541-066002fca229/volumes/kubernetes.io~projected/kube-api-access-pk6pt DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee07f20fc84768665db9cd20c3ca75747e807b661f06fc562e2a8b9ecd260375/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ea0819b8526091 MacAddress:ca:8b:4b:77:e0:15 Speed:10000 Mtu:8901} {Name:213bc0f1fb893fd MacAddress:52:f4:cf:a6:ee:a4 Speed:10000 Mtu:8901} {Name:24e06496d8ec7f2 MacAddress:ce:76:79:31:51:73 Speed:10000 Mtu:8901} {Name:285a57dac700951 MacAddress:d6:fb:fd:52:c8:ed Speed:10000 Mtu:8901} {Name:96c9c211a12896c MacAddress:5e:6f:8a:0a:a4:b4 Speed:10000 Mtu:8901} {Name:bf9594d715907a3 MacAddress:e2:af:6a:a2:3c:13 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:14:fe:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:27:b7:63:69:6d Speed:0 Mtu:8901} {Name:c1cce1905644710 MacAddress:06:ca:73:cd:3a:76 Speed:10000 Mtu:8901} {Name:c4200e6eedf98a3 MacAddress:0a:34:17:36:33:6c Speed:10000 Mtu:8901} {Name:da826a48d07d8ce MacAddress:f2:41:18:51:a2:84 Speed:10000 Mtu:8901} {Name:dbf22b0e710824e MacAddress:ba:76:42:55:bf:55 Speed:10000 Mtu:8901} {Name:e8fbc654ab5ad09 MacAddress:ca:f0:af:d4:e3:b2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:14:fe:95 Speed:0 Mtu:9001} {Name:fe3cd926e3b9ef1 MacAddress:02:97:02:0a:fe:47 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:d2:aa:21:cd:1e:e6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:88:6f:cc:bd:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 23:29:44.170954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.170779    6430 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 23:29:44.170954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.170826    6430 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 23:29:44.171240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171123    6430 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 23:29:44.171278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171146    6430 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-134.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 23:29:44.171322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171279    6430 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 23:29:44.171322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171287    6430 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 23:29:44.171322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171297    6430 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:29:44.171322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171317    6430 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 23:29:44.171468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171416    6430 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 23:29:44.171593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171583    6430 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 23:29:44.171690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171682    6430 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 23:29:44.171722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171718    6430 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 23:29:44.171752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171733    6430 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 23:29:44.171752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171743    6430 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 23:29:44.171752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171752    6430 kubelet.go:405] "Adding apiserver pod source"
Nov 03 23:29:44.171854 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.171760    6430 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 23:29:44.172511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.172492    6430 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 23:29:44.172511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.172510    6430 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:29:44.172949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.172931    6430 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 23:29:44.179759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.179740    6430 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 23:29:44.179759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.179756    6430 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 23:29:44.179921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.179879    6430 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 23:29:44.180111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180096    6430 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 23:29:44.180180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180136    6430 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 23:29:44.180277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180253    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 23:29:44.180277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180271    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180281    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180290    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180300    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180309    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180318    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180328    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180338    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180347    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180368    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 23:29:44.180407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180383    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 23:29:44.180839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180433    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 23:29:44.180839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.180444    6430 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 23:29:44.183929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.183913    6430 server.go:1263] "Started kubelet"
Nov 03 23:29:44.184441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.183985    6430 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 23:29:44.184441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.184002    6430 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 23:29:44.184441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.184170    6430 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 23:29:44.184618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.184591    6430 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 23:29:44.184508 ip-10-0-143-134 systemd[1]: Started Kubernetes Kubelet.
Nov 03 23:29:44.187875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.186640    6430 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 23:29:44.191100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191081    6430 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191103    6430 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191196    6430 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191207    6430 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191280    6430 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:33 +0000 UTC" deadline="2028-04-26 17:38:41.739741771 +0000 UTC"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191315    6430 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.191315    6430 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12954h8m57.548444032s"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.191590    6430 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-east-2.compute.internal\" not found"
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192422    6430 factory.go:55] Registering systemd factory
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192439    6430 factory.go:223] Registration of the systemd container factory successfully
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192693    6430 factory.go:153] Registering CRI-O factory
Nov 03 23:29:44.192738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192709    6430 factory.go:223] Registration of the crio container factory successfully
Nov 03 23:29:44.193117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192766    6430 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 23:29:44.193117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192786    6430 factory.go:103] Registering Raw factory
Nov 03 23:29:44.193117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.192801    6430 manager.go:1199] Started watching for new ooms in manager
Nov 03 23:29:44.193210 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.193143    6430 manager.go:319] Starting recovery of all containers
Nov 03 23:29:44.194703 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194663    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eeb5df1-2d70-4963-ab3f-503518320882" volumeName="kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194706    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194722    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c17332ae-befe-4495-8904-af7ad229d464" volumeName="kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194736    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eeb5df1-2d70-4963-ab3f-503518320882" volumeName="kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194750    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194763    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4" seLinuxMountContext=""
Nov 03 23:29:44.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194775    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a378b9cf-dc33-42fa-8c6e-de8497ba0c2f" volumeName="kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194787    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a378b9cf-dc33-42fa-8c6e-de8497ba0c2f" volumeName="kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194803    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194816    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194829    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" volumeName="kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194842    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a9a502-a9d0-448d-ae12-e7fa44dc3c2f" volumeName="kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194855    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194868    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194884    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194896    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" volumeName="kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194911    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f71c198-ea98-4678-8297-3d93e18fc042" volumeName="kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194923    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194935    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ddeb79-97e0-492a-a4d6-f50f96720fb7" volumeName="kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs" seLinuxMountContext=""
Nov 03 23:29:44.194946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194947    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a32217f-0822-43ae-8541-066002fca229" volumeName="kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194960    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194972    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f" volumeName="kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.194985    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ddeb79-97e0-492a-a4d6-f50f96720fb7" volumeName="kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195020    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcfd05a2-2670-445f-8d81-412fa24a1d60" volumeName="kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195033    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c17332ae-befe-4495-8904-af7ad229d464" volumeName="kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195046    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195059    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f71c198-ea98-4678-8297-3d93e18fc042" volumeName="kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195070    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195102    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195116    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195130    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195144    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" volumeName="kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195156    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5326b538-687b-4b39-b610-cf9b67362c02" volumeName="kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195169    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eeb5df1-2d70-4963-ab3f-503518320882" volumeName="kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195182    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9570168c-b9aa-455f-981e-33d57c4b5671" volumeName="kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195194    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcfd05a2-2670-445f-8d81-412fa24a1d60" volumeName="kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195206    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a32217f-0822-43ae-8541-066002fca229" volumeName="kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195219    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195231    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9570168c-b9aa-455f-981e-33d57c4b5671" volumeName="kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195245    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f" volumeName="kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195259    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c17332ae-befe-4495-8904-af7ad229d464" volumeName="kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195272    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195286    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a32217f-0822-43ae-8541-066002fca229" volumeName="kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195299    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c17332ae-befe-4495-8904-af7ad229d464" volumeName="kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195311    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c17332ae-befe-4495-8904-af7ad229d464" volumeName="kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195325    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d128c5e4-fb0c-486a-a7ef-fa8545746825" volumeName="kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195353    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195369    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04000032-9127-46a5-ba82-dcbb72b1e394" volumeName="kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195383    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195412    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ed813d-7b33-4539-8d6f-657a80430229" volumeName="kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195427    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ed813d-7b33-4539-8d6f-657a80430229" volumeName="kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert" seLinuxMountContext=""
Nov 03 23:29:44.195430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195439    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195455    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" volumeName="kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195478    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195492    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195504    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195527    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195539    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195554    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04000032-9127-46a5-ba82-dcbb72b1e394" volumeName="kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195569    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a9a502-a9d0-448d-ae12-e7fa44dc3c2f" volumeName="kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195582    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b40159c-efef-4f32-91d3-7f0d6a74a146" volumeName="kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195597    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195611    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195625    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195644    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195656    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f71c198-ea98-4678-8297-3d93e18fc042" volumeName="kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195668    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195681    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195694    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178f2de-cbfa-41db-9568-c52a97e0ec2d" volumeName="kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195707    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f71c198-ea98-4678-8297-3d93e18fc042" volumeName="kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195721    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a9a502-a9d0-448d-ae12-e7fa44dc3c2f" volumeName="kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195734    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5326b538-687b-4b39-b610-cf9b67362c02" volumeName="kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195746    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195759    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195771    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" volumeName="kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195783    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195796    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a378b9cf-dc33-42fa-8c6e-de8497ba0c2f" volumeName="kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195809    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e628017-63f5-4329-86b0-f07e84304f96" volumeName="kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195826    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a9a502-a9d0-448d-ae12-e7fa44dc3c2f" volumeName="kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195839    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195851    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89775551-7997-496e-84ac-55070f5b6665" volumeName="kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195864    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a378b9cf-dc33-42fa-8c6e-de8497ba0c2f" volumeName="kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195877    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195890    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfc458d-4b78-4b41-8ea4-79514fa4fe0f" volumeName="kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195902    6430 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" volumeName="kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g" seLinuxMountContext=""
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195957    6430 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 23:29:44.196491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.195968    6430 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 23:29:44.197679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.196875    6430 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:29:44.203248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.201694    6430 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:29:44.203248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.201822    6430 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:29:44.203248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.202834    6430 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 23:29:44.222129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.222071    6430 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 23:29:44.223541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.223518    6430 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 23:29:44.223541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.223545    6430 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 23:29:44.223741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.223565    6430 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 23:29:44.223741 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.223603    6430 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 23:29:44.229710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.228680    6430 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 23:29:44.237833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.237798    6430 generic.go:356] "Generic (PLEG): container finished" podID="026dcdf5-84bf-47d3-a512-6eef0ae86f5c" containerID="fdd5a8a3274fef1b8d2ac18595d11c91fe40ca5a47421c07bba0e253b2640f07" exitCode=0
Nov 03 23:29:44.275289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.275254    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab" exitCode=0
Nov 03 23:29:44.281300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281265    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="6f3cc23eaa70a1bec851e1a831072d395cccce593f90ac80aa5482afbfb28781" exitCode=0
Nov 03 23:29:44.281300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281301    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="2dedb8f986f450475096ca860f5833e5365ad547e779259e4eb3fd19fb439cd6" exitCode=0
Nov 03 23:29:44.281506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281312    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="73bc61213f0d2b2ffbbec27d3f33307e19f8ec3a11a1f5db2913cd12de374e35" exitCode=0
Nov 03 23:29:44.281506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281323    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="06d91eaa1da898ba7b78b280e17029adaf31305bdfe3cb46a1a90783f86f2d6f" exitCode=0
Nov 03 23:29:44.281506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281332    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="ed68516dd14011c41c31e2bf3eefb53cfe4e53d314b62555c7b3ea90c25b5dfe" exitCode=0
Nov 03 23:29:44.281506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.281346    6430 generic.go:356] "Generic (PLEG): container finished" podID="04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3" containerID="765c29ea950379321c39c246afee21b2edec2f6ed642b6867f4f72463ad0f166" exitCode=0
Nov 03 23:29:44.283962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.283939    6430 generic.go:356] "Generic (PLEG): container finished" podID="03308d9024657dde1775c95b6ad40488" containerID="a37043bc03df2d1a7c7c695cd19da0cf398dfccd66022d143e633a3a93390627" exitCode=0
Nov 03 23:29:44.304614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.304539    6430 manager.go:324] Recovery completed
Nov 03 23:29:44.323738 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.323697    6430 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 23:29:44.332056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332023    6430 cpu_manager.go:226] "Starting" policy="none"
Nov 03 23:29:44.332056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332048    6430 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332071    6430 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332216    6430 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332228    6430 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332246    6430 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332252    6430 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 23:29:44.332263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332257    6430 policy_none.go:50] "Start"
Nov 03 23:29:44.332475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332269    6430 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 23:29:44.332475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.332281    6430 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 23:29:44.333489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.333475    6430 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 23:29:44.333526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.333490    6430 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 23:29:44.333526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.333497    6430 policy_none.go:44] "Start"
Nov 03 23:29:44.339585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.339032    6430 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 23:29:44.345276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345259    6430 manager.go:341] "Starting Device Plugin manager"
Nov 03 23:29:44.345434 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.345291    6430 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 23:29:44.345434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345303    6430 server.go:87] "Starting device plugin registration server"
Nov 03 23:29:44.345653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345635    6430 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 23:29:44.345771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345651    6430 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 23:29:44.345834 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345806    6430 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 23:29:44.345936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345901    6430 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 23:29:44.345986 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345945    6430 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 23:29:44.345986 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.345953    6430 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 23:29:44.346096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.346032    6430 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:29:44.345934659Z","UUID":"c237beb8-b591-40dc-a8c6-afd28e98e59b","Handler":null,"Name":"","Endpoint":""}
Nov 03 23:29:44.348401 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.348367    6430 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 23:29:44.348504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.348433    6430 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 23:29:44.348741 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.348704    6430 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 23:29:44.446137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.446107    6430 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 23:29:44.448086 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.448066    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 23:29:44.448155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.448105    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 23:29:44.448155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.448116    6430 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 23:29:44.448215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.448181    6430 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.471873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.471845    6430 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.472005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.471919    6430 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.523857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.523795    6430 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"]
Nov 03 23:29:44.524537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524493    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerStarted","Data":"dc48f0316b9105fc766822ae687c743672026411c497c28036affeb086913953"}
Nov 03 23:29:44.524537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524524    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.524537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524536    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerDied","Data":"a37043bc03df2d1a7c7c695cd19da0cf398dfccd66022d143e633a3a93390627"}
Nov 03 23:29:44.524820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524550    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"03308d9024657dde1775c95b6ad40488","Type":"ContainerStarted","Data":"66703108734a82225ff5047e810d7ef2287a6a1ba61630934bf8bf115b970124"}
Nov 03 23:29:44.524820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524563    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"b6a44b6090d9fc279a15fb8a9a158ae7","Type":"ContainerStarted","Data":"abd2cb92d1b9d1ee277a186477de1fbd27ffbe24400759db5fa02bb4dc099b34"}
Nov 03 23:29:44.524820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524574    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" event={"ID":"b6a44b6090d9fc279a15fb8a9a158ae7","Type":"ContainerStarted","Data":"104e8ac9aa5a244e44dd88d1245bc2e020ed8efb760080c9b48182a8ed41cffa"}
Nov 03 23:29:44.524820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.524602    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.542926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.542903    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:29:44.543022 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.542946    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.546904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.546886    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:29:44.546976 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:44.546921    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.605315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.605240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.605315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.605278    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.605315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.605296    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.705992    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.706027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.706049    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.706108    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6a44b6090d9fc279a15fb8a9a158ae7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"b6a44b6090d9fc279a15fb8a9a158ae7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.706121    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:44.706228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:44.706108    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03308d9024657dde1775c95b6ad40488-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" (UID: \"03308d9024657dde1775c95b6ad40488\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:45.171941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.171907    6430 apiserver.go:52] "Watching apiserver"
Nov 03 23:29:45.200689 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.200652    6430 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 23:29:45.201316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.201286    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lz9rf","openshift-network-operator/iptables-alerter-wf22x","kube-system/konnectivity-agent-jnzl8","openshift-ingress-canary/ingress-canary-4wqzj","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb","openshift-multus/multus-additional-cni-plugins-hm9dt","openshift-dns/dns-default-ngzx9","kube-system/global-pull-secret-syncer-fpnrk","openshift-multus/multus-llrvd","openshift-image-registry/node-ca-6rhdg","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-8b55658f-66dwg","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal","openshift-dns/node-resolver-784mp","openshift-ovn-kubernetes/ovnkube-node-qts9p","openshift-cluster-node-tuning-operator/tuned-ffxlw","openshift-insights/insights-runtime-extractor-cswbt","openshift-monitoring/metrics-server-5b4f46b88d-r6wq8","openshift-network-diagnostics/network-check-target-zr4h4","openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj","openshift-monitoring/node-exporter-6cdd9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg","openshift-monitoring/monitoring-plugin-77f4f76594-zchvm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"]
Nov 03 23:29:45.206750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.206729    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 23:29:45.206853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.206771    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\""
Nov 03 23:29:45.210585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.210564    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 23:29:45.211313 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.211287    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\""
Nov 03 23:29:45.211928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.211903    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 23:29:45.212631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.212605    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 23:29:45.212738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.212726    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.217610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.217592    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.220059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220036    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 23:29:45.220161 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220121    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.220215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220192    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 23:29:45.220292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220276    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\""
Nov 03 23:29:45.220431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220406    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.220548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220531    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 23:29:45.220710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220690    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\""
Nov 03 23:29:45.220772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220710    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\""
Nov 03 23:29:45.220772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220712    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\""
Nov 03 23:29:45.220772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220734    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.220933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.220912    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 23:29:45.222570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222552    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.222774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222763    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 23:29:45.222774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222769    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 23:29:45.222905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222790    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.222905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222860    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.222905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222862    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.222905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.222869    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.223412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223381    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 23:29:45.223446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223411    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 23:29:45.223446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223415    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.223510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223486    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 23:29:45.223552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223507    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.223552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223527    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 23:29:45.223552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223544    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\""
Nov 03 23:29:45.223653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223551    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\""
Nov 03 23:29:45.223653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.223548    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 23:29:45.225838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.225819    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 23:29:45.225838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.225834    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\""
Nov 03 23:29:45.225982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.225826    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.225982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.225838    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.225982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.225846    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226020    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226064    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226081    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226082    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226109    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 23:29:45.226185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.226142    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 23:29:45.228550 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228536    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.228607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228560    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 23:29:45.228607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228585    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 23:29:45.228712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228633    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 23:29:45.228712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228680    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 23:29:45.228712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228695    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 23:29:45.228712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228704    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.228712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228712    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228719    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228747    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228784    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xtxcc\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228786    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228787    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.228893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.228789    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 23:29:45.231064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231053    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 23:29:45.231223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231212    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 23:29:45.231280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231251    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fw76h\""
Nov 03 23:29:45.231329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231252    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 23:29:45.231329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231287    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\""
Nov 03 23:29:45.231329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231295    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 23:29:45.231329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231309    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2dmvx\""
Nov 03 23:29:45.231468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.231332    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8sbct\""
Nov 03 23:29:45.233152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233134    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mlhdz\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233152    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233158    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233153    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233170    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233172    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233144    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 23:29:45.233224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233205    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 23:29:45.233509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233270    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 23:29:45.233509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.233332    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\""
Nov 03 23:29:45.238160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238146    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 23:29:45.238239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238170    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 23:29:45.238239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238207    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 23:29:45.238239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238213    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 23:29:45.238351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238255    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 23:29:45.238351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238320    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 23:29:45.238351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238310    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 23:29:45.238959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.238945    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 23:29:45.239788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.239774    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e0s0ptbrqipi7\""
Nov 03 23:29:45.240895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.240881    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxkfg\""
Nov 03 23:29:45.241695 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.241678    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3lqhglk32h6u6\""
Nov 03 23:29:45.241746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.241725    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\""
Nov 03 23:29:45.241746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.241731    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2cj7f\""
Nov 03 23:29:45.241823 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.241799    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 23:29:45.289652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.289622    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:45.289830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.289744    6430 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:45.293701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.293679    6430 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 23:29:45.307448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307427    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:29:45.307528 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:45.307464    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:45.307817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.307851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307826    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.307885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307858    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.307918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307899    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcjt\" (UniqueName: \"kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.307948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307934    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.307977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307955    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.308016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307973    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:29:45.308016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.307997    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.308111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308063    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.308111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308096    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.308245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm26\" (UniqueName: \"kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:45.308245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308130    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-metrics-client-ca\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.308245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308243    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.308344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308254    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9bw\" (UniqueName: \"kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.308344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308314    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.308523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308340    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-agent-certs\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:29:45.308523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308402    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308445    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.308523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308490    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308531    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308576    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-config\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308591    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xgr\" (UniqueName: \"kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308645    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308670    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308708    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.308739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308714    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.309045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308747    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzrj\" (UniqueName: \"kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.309045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308861    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.309260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.308905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c17332ae-befe-4495-8904-af7ad229d464-ovn-node-metrics-cert\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.309337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309285    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.309414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309360    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.309467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309427    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.309542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309467    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:45.309542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309473    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f71c198-ea98-4678-8297-3d93e18fc042-config-volume\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.309542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309495    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-etc-tuned\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.309542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309504    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwzf\" (UniqueName: \"kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.309786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309628    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.309786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309683    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.309786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309721    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.309786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309727    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5326b538-687b-4b39-b610-cf9b67362c02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:45.309786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309744    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-client-certs\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309858    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309960    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309984    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-tls\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.309994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310022    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310025    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310034    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.310089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310073    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310171    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310223    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310293    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310352    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310366    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310444    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7v7\" (UniqueName: \"kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.310484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310637    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310694    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310744    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310771    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310814    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.310923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310947    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311028    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.310954    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311132    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311167    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311176    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-ovnkube-script-lib\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311212    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.311265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311244    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311283    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311314    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311362    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311462    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcfd05a2-2670-445f-8d81-412fa24a1d60-metrics-certs\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311544    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311592    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311625    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311672    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpsp\" (UniqueName: \"kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.311717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311704    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbh7\" (UniqueName: \"kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311742    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311774    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311820    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311849    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311893    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311921    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311958    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.311990    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.312152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312146    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.312662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312191    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.312662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312227    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2gj\" (UniqueName: \"kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:45.312662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312264    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.312662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312546    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.312866 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312680    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.312866 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312766    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.312968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.313023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-cni-binary-copy\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313204    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313339    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f178f2de-cbfa-41db-9568-c52a97e0ec2d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.312271    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f71c198-ea98-4678-8297-3d93e18fc042-metrics-tls\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313455    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-textfile\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313488    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313523    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313553    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313624    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.313671    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314023    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-grpc-tls\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.314055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314019    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314117    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-tmp\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314156    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314190    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314327    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314369    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314460    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314509    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c17332ae-befe-4495-8904-af7ad229d464-env-overrides\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314578    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314588    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddeb79-97e0-492a-a4d6-f50f96720fb7-konnectivity-ca\") pod \"konnectivity-agent-jnzl8\" (UID: \"b0ddeb79-97e0-492a-a4d6-f50f96720fb7\") " pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314609    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.314683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314673    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.315220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314739    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.315220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.314810    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.315220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315163    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.315220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315195    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.315220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315164    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-metrics-server-audit-profiles\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315257    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315266    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315272    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315287    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315302    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9570168c-b9aa-455f-981e-33d57c4b5671-iptables-alerter-script\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315329    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315403    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315434    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.315482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315482    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315485    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-serviceca\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315570    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315600    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6pt\" (UniqueName: \"kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71c198-ea98-4678-8297-3d93e18fc042-tmp-dir\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315631    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315647    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315745    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315776    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d128c5e4-fb0c-486a-a7ef-fa8545746825-original-pull-secret\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315792    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-secret-metrics-server-tls\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315792    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd87g\" (UniqueName: \"kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.315896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315871    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315942    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.315994    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316007    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316035    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316040    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316099    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316116    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316114    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a32217f-0822-43ae-8541-066002fca229-tmp\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316132    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316164    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316201    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5eeb5df1-2d70-4963-ab3f-503518320882-multus-daemon-config\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316207    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316242    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316253    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316333    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ed813d-7b33-4539-8d6f-657a80430229-cert\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:45.316371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316344    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5h2p\" (UniqueName: \"kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316383    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89775551-7997-496e-84ac-55070f5b6665-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316445    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316476    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316501    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316520    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316521    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-metrics-client-ca\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316538    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6k2\" (UniqueName: \"kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316574    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316588    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316589    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04000032-9127-46a5-ba82-dcbb72b1e394-tmp-dir\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316623    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316660    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316685    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316688    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316727    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-data-volume\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316804    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316806    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgpzz\" (UniqueName: \"kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316845    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316870    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316895    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89x9\" (UniqueName: \"kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316920    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.316968    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317073    6430 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317075    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89775551-7997-496e-84ac-55070f5b6665-client-ca-bundle\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317076    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rjq\" (UniqueName: \"kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:45.317089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317108    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89775551-7997-496e-84ac-55070f5b6665-audit-log\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317133    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f178f2de-cbfa-41db-9568-c52a97e0ec2d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317159    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317265    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:29:45.317290    6430 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-east-2.compute.internal"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317412    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.318090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.317436    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.325946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.325918    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9bw\" (UniqueName: \"kubernetes.io/projected/04000032-9127-46a5-ba82-dcbb72b1e394-kube-api-access-4z9bw\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.330200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.330181    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcjt\" (UniqueName: \"kubernetes.io/projected/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-kube-api-access-xdcjt\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.330981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.330965    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xgr\" (UniqueName: \"kubernetes.io/projected/ecfc458d-4b78-4b41-8ea4-79514fa4fe0f-kube-api-access-t5xgr\") pod \"thanos-querier-8b55658f-66dwg\" (UID: \"ecfc458d-4b78-4b41-8ea4-79514fa4fe0f\") " pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:45.332708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.332693    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsv5\" (UniqueName: \"kubernetes.io/projected/6b40159c-efef-4f32-91d3-7f0d6a74a146-kube-api-access-wjsv5\") pod \"network-check-target-zr4h4\" (UID: \"6b40159c-efef-4f32-91d3-7f0d6a74a146\") " pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:29:45.343819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.343802    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm26\" (UniqueName: \"kubernetes.io/projected/b9ed813d-7b33-4539-8d6f-657a80430229-kube-api-access-qfm26\") pod \"ingress-canary-4wqzj\" (UID: \"b9ed813d-7b33-4539-8d6f-657a80430229\") " pod="openshift-ingress-canary/ingress-canary-4wqzj"
Nov 03 23:29:45.344635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.344619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/9570168c-b9aa-455f-981e-33d57c4b5671-kube-api-access-t65tt\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.344863 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.344847    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwzf\" (UniqueName: \"kubernetes.io/projected/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-kube-api-access-grwzf\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.345339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.345319    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6pt\" (UniqueName: \"kubernetes.io/projected/5a32217f-0822-43ae-8541-066002fca229-kube-api-access-pk6pt\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.345960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.345942    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"alertmanager-main-0\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:29:45.347988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.347970    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5h2p\" (UniqueName: \"kubernetes.io/projected/f178f2de-cbfa-41db-9568-c52a97e0ec2d-kube-api-access-k5h2p\") pod \"kube-state-metrics-5dc8d7d89b-cjpmb\" (UID: \"f178f2de-cbfa-41db-9568-c52a97e0ec2d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cjpmb"
Nov 03 23:29:45.349942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.349924    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzrj\" (UniqueName: \"kubernetes.io/projected/10a9a502-a9d0-448d-ae12-e7fa44dc3c2f-kube-api-access-6bzrj\") pod \"openshift-state-metrics-7bd7489957-ts8bj\" (UID: \"10a9a502-a9d0-448d-ae12-e7fa44dc3c2f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ts8bj"
Nov 03 23:29:45.350497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.350480    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rjq\" (UniqueName: \"kubernetes.io/projected/bcfd05a2-2670-445f-8d81-412fa24a1d60-kube-api-access-v4rjq\") pod \"network-metrics-daemon-lz9rf\" (UID: \"bcfd05a2-2670-445f-8d81-412fa24a1d60\") " pod="openshift-multus/network-metrics-daemon-lz9rf"
Nov 03 23:29:45.350609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.350592    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbh7\" (UniqueName: \"kubernetes.io/projected/89775551-7997-496e-84ac-55070f5b6665-kube-api-access-rwbh7\") pod \"metrics-server-5b4f46b88d-r6wq8\" (UID: \"89775551-7997-496e-84ac-55070f5b6665\") " pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:45.351130 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.351087    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpsp\" (UniqueName: \"kubernetes.io/projected/5eeb5df1-2d70-4963-ab3f-503518320882-kube-api-access-tkpsp\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.351524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.351509    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgpzz\" (UniqueName: \"kubernetes.io/projected/0e628017-63f5-4329-86b0-f07e84304f96-kube-api-access-tgpzz\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.351566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.351552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89x9\" (UniqueName: \"kubernetes.io/projected/c17332ae-befe-4495-8904-af7ad229d464-kube-api-access-h89x9\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.351678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.351663    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2gj\" (UniqueName: \"kubernetes.io/projected/5326b538-687b-4b39-b610-cf9b67362c02-kube-api-access-6j2gj\") pod \"monitoring-plugin-77f4f76594-zchvm\" (UID: \"5326b538-687b-4b39-b610-cf9b67362c02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:45.353822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.353805    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7v7\" (UniqueName: \"kubernetes.io/projected/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-kube-api-access-pg7v7\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.355253 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.355238    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd87g\" (UniqueName: \"kubernetes.io/projected/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-kube-api-access-vd87g\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.356723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.356707    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6k2\" (UniqueName: \"kubernetes.io/projected/0f71c198-ea98-4678-8297-3d93e18fc042-kube-api-access-gg6k2\") pod \"dns-default-ngzx9\" (UID: \"0f71c198-ea98-4678-8297-3d93e18fc042\") " pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:45.417978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.417946    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.417978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.417978    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418001    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418044    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-kubelet\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418049    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f-host\") pod \"node-ca-6rhdg\" (UID: \"a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f\") " pod="openshift-image-registry/node-ca-6rhdg"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418098    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418106    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.418136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418133    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-socket-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418178    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-socket-dir-parent\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418179    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418201    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418210    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-systemd\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418219    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418235    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418250    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-etc-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418250    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418257    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-hostroot\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418277    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418304    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418308    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-netd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418323    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418277    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a378b9cf-dc33-42fa-8c6e-de8497ba0c2f-crio-socket\") pod \"insights-runtime-extractor-cswbt\" (UID: \"a378b9cf-dc33-42fa-8c6e-de8497ba0c2f\") " pod="openshift-insights/insights-runtime-extractor-cswbt"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418342    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418303    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-k8s-cni-cncf-io\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418363    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418368    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.418379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418414    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418367    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-systemd\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418452    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-etc-kubernetes\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418464    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-sys\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418477    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418501    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418523    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418531    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysctl-conf\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-log-socket\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418564    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-conf-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418570    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418591    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418565    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-netns\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418606    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418625    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418630    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-slash\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418655    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418658    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-run\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418670    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418677    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418682    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-kubelet\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418685    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418700    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418707    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-var-lib-openvswitch\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418713    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418727    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418757    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418794    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418797    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-sys\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418807    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-kubelet-config\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418798    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-system-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418820    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418776    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-cnibin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418833    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-node-exporter-wtmp\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418812    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418848    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418849    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-run-multus-certs\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418851    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-host\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418873    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418873    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3-cnibin\") pod \"multus-additional-cni-plugins-hm9dt\" (UID: \"04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3\") " pod="openshift-multus/multus-additional-cni-plugins-hm9dt"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418882    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418909    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-multus\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418915    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418938    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/026dcdf5-84bf-47d3-a512-6eef0ae86f5c-root\") pod \"node-exporter-6cdd9\" (UID: \"026dcdf5-84bf-47d3-a512-6eef0ae86f5c\") " pod="openshift-monitoring/node-exporter-6cdd9"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418950    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418972    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.418987    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419013    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419015    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04000032-9127-46a5-ba82-dcbb72b1e394-hosts-file\") pod \"node-resolver-784mp\" (UID: \"04000032-9127-46a5-ba82-dcbb72b1e394\") " pod="openshift-dns/node-resolver-784mp"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419039    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419052    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-systemd-units\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419061    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419061    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-run-ovn\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419069    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419078    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d128c5e4-fb0c-486a-a7ef-fa8545746825-dbus\") pod \"global-pull-secret-syncer-fpnrk\" (UID: \"d128c5e4-fb0c-486a-a7ef-fa8545746825\") " pod="kube-system/global-pull-secret-syncer-fpnrk"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419090    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419097    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-registration-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419104    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-run-netns\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419171    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419184    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-modprobe-d\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.419138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419185    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419196    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-device-dir\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419203    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9570168c-b9aa-455f-981e-33d57c4b5671-host-slash\") pod \"iptables-alerter-wf22x\" (UID: \"9570168c-b9aa-455f-981e-33d57c4b5671\") " pod="openshift-network-operator/iptables-alerter-wf22x"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419216    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419236    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419239    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-host-var-lib-cni-bin\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419262    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419276    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419293    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419316    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-node-log\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419327    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419321    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-sysconfig\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5eeb5df1-2d70-4963-ab3f-503518320882-multus-cni-dir\") pod \"multus-llrvd\" (UID: \"5eeb5df1-2d70-4963-ab3f-503518320882\") " pod="openshift-multus/multus-llrvd"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c17332ae-befe-4495-8904-af7ad229d464-host-cni-bin\") pod \"ovnkube-node-qts9p\" (UID: \"c17332ae-befe-4495-8904-af7ad229d464\") " pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419370    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419377    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e628017-63f5-4329-86b0-f07e84304f96-sys-fs\") pod \"aws-ebs-csi-driver-node-wn8cg\" (UID: \"0e628017-63f5-4329-86b0-f07e84304f96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn8cg"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419415    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419438    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-etc-kubernetes\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419416    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-lib-modules\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.421018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.419458    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a32217f-0822-43ae-8541-066002fca229-var-lib-kubelet\") pod \"tuned-ffxlw\" (UID: \"5a32217f-0822-43ae-8541-066002fca229\") " pod="openshift-cluster-node-tuning-operator/tuned-ffxlw"
Nov 03 23:29:45.504946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:45.504926    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:29:46.950846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:46.950782    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ngzx9"
Nov 03 23:29:47.505416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:47.505369    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8b55658f-66dwg"
Nov 03 23:29:47.776606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:47.776525    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:48.108860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:48.108827    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:29:48.307556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:48.307522    6430 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:29:48.827952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:48.827926    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zchvm"
Nov 03 23:29:49.312547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:49.312510    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr4h4" event={"ID":"6b40159c-efef-4f32-91d3-7f0d6a74a146","Type":"ContainerStarted","Data":"61053a0e256d3f3c5c75b1e4396faddc92bbc8d29a7179eacec0a5c02d2ede36"}
Nov 03 23:29:55.239288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:55.239217    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b4f46b88d-r6wq8"
Nov 03 23:29:55.253729 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:55.253700    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnzl8"
Nov 03 23:29:55.280491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:29:55.280465    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:30:00.005230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005187    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:00.005739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005699    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-web" containerID="cri-o://704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e" gracePeriod=120
Nov 03 23:30:00.005973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005677    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="alertmanager" containerID="cri-o://35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf" gracePeriod=120
Nov 03 23:30:00.006047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005698    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-metric" containerID="cri-o://aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0" gracePeriod=120
Nov 03 23:30:00.006089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005710    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy" containerID="cri-o://2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f" gracePeriod=120
Nov 03 23:30:00.006130 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005730    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="config-reloader" containerID="cri-o://eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d" gracePeriod=120
Nov 03 23:30:00.006178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.005765    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="prom-label-proxy" containerID="cri-o://48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd" gracePeriod=120
Nov 03 23:30:00.248686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.248662    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.346153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346119    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346167    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346198    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346222    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346244    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346280    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346317    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346383    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346437    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346469    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346518    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346649    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") pod \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\" (UID: \"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90\") "
Nov 03 23:30:00.346968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.346899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:30:00.347605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.347337    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:30:00.348302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.348267    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:30:00.349824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.349779    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.351071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.350982    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:30:00.351071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.351020    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.351230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.351173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.351336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.351293    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.351468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.351365    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.351551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.351529    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:30:00.352105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.352084    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "kube-api-access-55bz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:30:00.356133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.356108    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.357631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357606    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd" exitCode=0
Nov 03 23:30:00.357631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357627    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0" exitCode=0
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357633    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f" exitCode=0
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357642    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e" exitCode=0
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357647    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d" exitCode=0
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357653    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf" exitCode=0
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357686    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.357771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357691    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"}
Nov 03 23:30:00.357963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357802    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"}
Nov 03 23:30:00.357963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357855    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"}
Nov 03 23:30:00.357963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357872    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.357963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357898    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"}
Nov 03 23:30:00.357963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357941    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"}
Nov 03 23:30:00.358146 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.357987    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"}
Nov 03 23:30:00.358146 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.358031    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90","Type":"ContainerDied","Data":"213bc0f1fb893fd81b44abd7f5f7fdb78d20feb1118b3fa90be93257798ab12b"}
Nov 03 23:30:00.362502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.362471    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config" pod "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" (UID: "7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:30:00.367638 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.367614    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.374878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.374856    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.381687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.381666    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.388452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.388432    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.395033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.395014    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.401896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.401881    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.408889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.408873    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.409158 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.409136    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.409209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409170    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.409209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409204    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.409478 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.409453    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.409533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409479    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.409533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409494    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.409712 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.409694    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.409774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409721    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.409774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409744    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.409982 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.409963    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.410038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.409988    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.410038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410004    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.410217 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.410200    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.410278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410224    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.410278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410243    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.410450 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.410436    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.410487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410453    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.410487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410465    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.410707 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:00.410690    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.410766 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410716    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.410766 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410734    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.410966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410950    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.411009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.410966    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.411168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411150    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.411209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411170    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.411362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411347    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.411435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411361    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.411597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411583    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.411597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411596    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.411816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411797    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.411882 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.411817    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.412034 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412019    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.412087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412033    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.412214 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412200    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.412251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412213    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.412453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412436    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.412453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412453    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.412658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412639    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.412726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412658    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.412864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412850    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.412912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.412864    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.413047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413034    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.413047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413046    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.413237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413220    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.413302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413237    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.413456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413442    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.413504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413455    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.413686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413668    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.413754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413686    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.413887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413872    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.413938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.413887    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.414067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414050    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.414132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414067    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.414272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414258    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.414320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414271    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.414485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414466    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.414558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414486    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.414687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414669    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.414732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414687    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.414887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414870    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.414955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.414887    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.415083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415066    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.415132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415083    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.415266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415248    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.415330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415266    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.415482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415468    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.415482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415482    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.415684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415664    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.415726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415685    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.415863 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415848    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.415899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.415862    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.416048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416028    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.416086 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416049    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.416233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416216    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.416273 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416234    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.416464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416445    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.416520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416465    6430 scope.go:122] "RemoveContainer" containerID="48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"
Nov 03 23:30:00.416657 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416642    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd"} err="failed to get container status \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": rpc error: code = NotFound desc = could not find container \"48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd\": container with ID starting with 48172b640832744cf9697cd018b3851390d4e543768b84782a57c05f58562ebd not found: ID does not exist"
Nov 03 23:30:00.416657 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416656    6430 scope.go:122] "RemoveContainer" containerID="aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"
Nov 03 23:30:00.416864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416848    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0"} err="failed to get container status \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": rpc error: code = NotFound desc = could not find container \"aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0\": container with ID starting with aa3bef76d2c3fd1b53464d2b99fa04b2434e672ac91dac617a2692db8eabb6a0 not found: ID does not exist"
Nov 03 23:30:00.416903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.416864    6430 scope.go:122] "RemoveContainer" containerID="2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"
Nov 03 23:30:00.417041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417027    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f"} err="failed to get container status \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": rpc error: code = NotFound desc = could not find container \"2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f\": container with ID starting with 2e28749289b99bd81b4b049e4b75bfb668ab5129b43a0b06e1a4dff0339aaa8f not found: ID does not exist"
Nov 03 23:30:00.417113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417042    6430 scope.go:122] "RemoveContainer" containerID="704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"
Nov 03 23:30:00.417228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417211    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e"} err="failed to get container status \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": rpc error: code = NotFound desc = could not find container \"704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e\": container with ID starting with 704e069a73b772f8a03cac1f88980edb6a50add58b1c9c65842e88ef2e2d520e not found: ID does not exist"
Nov 03 23:30:00.417268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417229    6430 scope.go:122] "RemoveContainer" containerID="eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"
Nov 03 23:30:00.417445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417423    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d"} err="failed to get container status \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": rpc error: code = NotFound desc = could not find container \"eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d\": container with ID starting with eee6564d34cf7acf6f1caba7e6d169913dadfb1f36dc37cdc142ac00b929984d not found: ID does not exist"
Nov 03 23:30:00.417488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417446    6430 scope.go:122] "RemoveContainer" containerID="35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"
Nov 03 23:30:00.417642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417623    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf"} err="failed to get container status \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": rpc error: code = NotFound desc = could not find container \"35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf\": container with ID starting with 35366a1acf66e700584236e0e889a5423f511e78086197ea3109adaab6973ebf not found: ID does not exist"
Nov 03 23:30:00.417696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417642    6430 scope.go:122] "RemoveContainer" containerID="c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"
Nov 03 23:30:00.417834 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.417814    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab"} err="failed to get container status \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": rpc error: code = NotFound desc = could not find container \"c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab\": container with ID starting with c3cbc6df5bd9b1c6e68c163e53b7d0020ea59961fcf7240301bdd342a29db9ab not found: ID does not exist"
Nov 03 23:30:00.448257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448228    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-metrics-client-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448253    6430 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-main-db\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448265    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448299    6430 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-web-config\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448308    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bz4\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-kube-api-access-55bz4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448318    6430 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448328    6430 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-tls-assets\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448337    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-main-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448346    6430 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448355    6430 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-cluster-tls-config\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448364    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448372    6430 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-config-out\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.448447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.448381    6430 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:30:00.687252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.687198    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:00.694473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.694449    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:00.726845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.726819    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:00.727115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727102    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-metric"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727116    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-metric"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727126    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727131    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727146    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="config-reloader"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727150    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="config-reloader"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727161    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-web"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727166    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-web"
Nov 03 23:30:00.727178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727178    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="init-config-reloader"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727182    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="init-config-reloader"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727192    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="prom-label-proxy"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727197    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="prom-label-proxy"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727207    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="alertmanager"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727211    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="alertmanager"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727255    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-web"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727262    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="init-config-reloader"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727268    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="prom-label-proxy"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727276    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="alertmanager"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727282    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy-metric"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727288    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="kube-rbac-proxy"
Nov 03 23:30:00.727446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.727295    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" containerName="config-reloader"
Nov 03 23:30:00.731013 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.730996    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.745288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.745262    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 23:30:00.762814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.762785    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:00.822432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822369    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 23:30:00.822432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822419    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 23:30:00.822432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822436    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822432    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822438    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822370    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822488    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822566    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 23:30:00.822718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.822645    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2dmvx\""
Nov 03 23:30:00.851024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.850989    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851059    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851119    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggvm\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-kube-api-access-pggvm\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851209    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851263    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851338    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851380    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851491    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-out\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851535    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851559    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-web-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851585    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.851722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.851663    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952465    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952589    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952625    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952655    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952794    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggvm\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-kube-api-access-pggvm\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952843    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.952878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952868    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952907    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952909    6430 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 23:30:00.953026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952944    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.952981    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-out\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.953006    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.953027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-web-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.953044    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.953066    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.953985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.953662    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.954435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.954112    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.955658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.955634    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-out\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.955759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.955717    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.955829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.955754    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.955909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.955878    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.956242 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.956221    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.956339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.956288    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.956584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.956562    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.956930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.956912    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-web-config\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.957478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.957462    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:00.966783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:00.966764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggvm\" (UniqueName: \"kubernetes.io/projected/a39f5e0e-2391-4581-ad5f-ec7544e32bdb-kube-api-access-pggvm\") pod \"alertmanager-main-0\" (UID: \"a39f5e0e-2391-4581-ad5f-ec7544e32bdb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:01.047874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:01.047840    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 23:30:01.337401 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:01.337347    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 23:30:01.340476 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:30:01.340442    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f5e0e_2391_4581_ad5f_ec7544e32bdb.slice/crio-ca89899f02837487a2161f4fd8577bdcf50fb639a20b644ea74d01a7cbf6b67f WatchSource:0}: Error finding container ca89899f02837487a2161f4fd8577bdcf50fb639a20b644ea74d01a7cbf6b67f: Status 404 returned error can't find the container with id ca89899f02837487a2161f4fd8577bdcf50fb639a20b644ea74d01a7cbf6b67f
Nov 03 23:30:01.363160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:01.363104    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"ca89899f02837487a2161f4fd8577bdcf50fb639a20b644ea74d01a7cbf6b67f"}
Nov 03 23:30:02.231252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:02.231215    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90" path="/var/lib/kubelet/pods/7b9d3aa6-8cfa-46fd-933e-28afd3cc3f90/volumes"
Nov 03 23:30:02.371578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:02.371533    6430 generic.go:356] "Generic (PLEG): container finished" podID="a39f5e0e-2391-4581-ad5f-ec7544e32bdb" containerID="d5ca5d726851b1ee3dcc2f27c79ce2ba9d0d6a1a6862d40cbc99fdd2ca87024c" exitCode=0
Nov 03 23:30:02.371769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:02.371601    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerDied","Data":"d5ca5d726851b1ee3dcc2f27c79ce2ba9d0d6a1a6862d40cbc99fdd2ca87024c"}
Nov 03 23:30:02.806768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:02.806739    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qts9p"
Nov 03 23:30:03.378868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.378835    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"99661649e095817a74d4ef4790003c69aeaf439c9c50e8f8e00d8934fa27927f"}
Nov 03 23:30:03.379267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.378904    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"3222af3f8a33df6a2c65339292e8d3f6522bb4ac2f5649dae7b1f2973865ae82"}
Nov 03 23:30:03.379267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.378945    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"3ad75addd1abecab6a510f6332acdaa1f1a6eddefbf540eb8f03aff7727f541a"}
Nov 03 23:30:03.379267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.378970    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"09c2889176f74571b94a8b7430d89b42cc79aaac1341fac026fa876ffbd6d6b6"}
Nov 03 23:30:03.379267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.378991    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"e2634eb3770283e51701b7d42d5c0327f604479594b35449897be973d74d133d"}
Nov 03 23:30:03.379267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.379025    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a39f5e0e-2391-4581-ad5f-ec7544e32bdb","Type":"ContainerStarted","Data":"d9f50da7d4d072ed39e0b93edaa1c56fa392f24f38223d22ea11e533d4fb3422"}
Nov 03 23:30:03.418897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:03.418819    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.405027059 podStartE2EDuration="3.418802837s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.775778ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:03.415744223 +0000 UTC m=+19.306806157" watchObservedRunningTime="2026-11-03 23:30:03.418802837 +0000 UTC m=+19.309864831"
Nov 03 23:30:04.829043 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:04.829007    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:30:25.283792 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:30:25.283764    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zr4h4"
Nov 03 23:30:34.830585 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:30:34.830540    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:31:04.830300 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:31:04.830259    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:31:34.829496 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:31:34.829458    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:32:04.829652 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:32:04.829615    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:32:26.228957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.228925    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:32:26.232009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.231991    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.241787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.241761    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:32:26.246332 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246312    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 23:32:26.246462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246378    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 23:32:26.246462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246431    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 23:32:26.246588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246496    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 23:32:26.246588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246553    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 23:32:26.246660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246590    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q44ht\""
Nov 03 23:32:26.246824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.246810    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 23:32:26.323442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323443    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323563    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnzm\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323629    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323672    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.323895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.323768    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424443    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424483    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424547    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnzm\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424577    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424610    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.424727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.424646    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.425234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.425202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.426997 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.426971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.427097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.427003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.427097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.427023    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.427168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.427134    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.438173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.438144    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.438411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.438378    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnzm\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\") pod \"istiod-openshift-gateway-7876bfcb74-5t59k\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.545612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.545542    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:26.822797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.822724    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:32:26.826114 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:32:26.826084    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99b3523_8e07_4982_bb6f_35adaec7cf68.slice/crio-f73f125916f558fc72461497a4ccf18abe02c591f7f8be037b381efd34ae485b WatchSource:0}: Error finding container f73f125916f558fc72461497a4ccf18abe02c591f7f8be037b381efd34ae485b: Status 404 returned error can't find the container with id f73f125916f558fc72461497a4ccf18abe02c591f7f8be037b381efd34ae485b
Nov 03 23:32:26.862840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:26.862804    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" event={"ID":"f99b3523-8e07-4982-bb6f-35adaec7cf68","Type":"ContainerStarted","Data":"f73f125916f558fc72461497a4ccf18abe02c591f7f8be037b381efd34ae485b"}
Nov 03 23:32:29.962233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:29.962197    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:32:29.962574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:29.962264    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:32:30.879409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:30.879351    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" event={"ID":"f99b3523-8e07-4982-bb6f-35adaec7cf68","Type":"ContainerStarted","Data":"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"}
Nov 03 23:32:30.879588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:30.879486    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:30.909171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:30.909126    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" podStartSLOduration=1.7757190189999998 podStartE2EDuration="4.9091123s" totalImagesPullingTime="3.133393281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:30.906906677 +0000 UTC m=+166.797968647" watchObservedRunningTime="2026-11-03 23:32:30.9091123 +0000 UTC m=+166.800174231"
Nov 03 23:32:31.884951 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:32:31.884919    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:32:34.834928 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:32:34.834893    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:33:04.828977 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:33:04.828896    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:33:22.797280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.797241    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"]
Nov 03 23:33:22.805503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.805480    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.847408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.847360    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"]
Nov 03 23:33:22.871577 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.871549    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 23:33:22.871720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.871591    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 23:33:22.872338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.872323    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 23:33:22.872433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.872338    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bc7w7\""
Nov 03 23:33:22.872433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.872365    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 23:33:22.872572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.872560    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 23:33:22.958855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.958818    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.958855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.958857    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.959060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.958893    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.959060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.958969    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm548\" (UniqueName: \"kubernetes.io/projected/82a246af-db2d-4e41-9c03-a36945082ffd-kube-api-access-jm548\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.959060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.958999    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:22.959060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:22.959024    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.059838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059753    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.059838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059791    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.059838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059824    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.060117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059857    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm548\" (UniqueName: \"kubernetes.io/projected/82a246af-db2d-4e41-9c03-a36945082ffd-kube-api-access-jm548\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.060117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059892    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.060117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.059926    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.060434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.060380    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.062292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.062268    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a246af-db2d-4e41-9c03-a36945082ffd-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.062610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.062589    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.062687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.062668    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.062807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.062786    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82a246af-db2d-4e41-9c03-a36945082ffd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.082786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.082759    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm548\" (UniqueName: \"kubernetes.io/projected/82a246af-db2d-4e41-9c03-a36945082ffd-kube-api-access-jm548\") pod \"opendatahub-operator-controller-manager-775bfbdf99-lhbhz\" (UID: \"82a246af-db2d-4e41-9c03-a36945082ffd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.119525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.119502    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:23.462357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:23.462324    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"]
Nov 03 23:33:23.463503 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:33:23.463475    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a246af_db2d_4e41_9c03_a36945082ffd.slice/crio-d3290e70bd676b47eae5023b8f2f22c5174c1b13485620351fdd0f609ea672d3 WatchSource:0}: Error finding container d3290e70bd676b47eae5023b8f2f22c5174c1b13485620351fdd0f609ea672d3: Status 404 returned error can't find the container with id d3290e70bd676b47eae5023b8f2f22c5174c1b13485620351fdd0f609ea672d3
Nov 03 23:33:24.070022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:24.069975    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" event={"ID":"82a246af-db2d-4e41-9c03-a36945082ffd","Type":"ContainerStarted","Data":"d3290e70bd676b47eae5023b8f2f22c5174c1b13485620351fdd0f609ea672d3"}
Nov 03 23:33:27.083489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:27.083456    6430 generic.go:356] "Generic (PLEG): container finished" podID="82a246af-db2d-4e41-9c03-a36945082ffd" containerID="6221551268c87010c33b386505b9d3f4e8dcaec431fad97edd52342fe3131e79" exitCode=0
Nov 03 23:33:27.083861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:27.083509    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" event={"ID":"82a246af-db2d-4e41-9c03-a36945082ffd","Type":"ContainerDied","Data":"6221551268c87010c33b386505b9d3f4e8dcaec431fad97edd52342fe3131e79"}
Nov 03 23:33:28.089582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:28.089545    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" event={"ID":"82a246af-db2d-4e41-9c03-a36945082ffd","Type":"ContainerStarted","Data":"c98ac11da5e8df0746ec649cd8bbece9197648eb2743730a448eeca76743d564"}
Nov 03 23:33:28.090151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:28.089699    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:28.197696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:28.197636    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" podStartSLOduration=2.212539151 podStartE2EDuration="6.197621137s" totalImagesPullingTime="3.427637038s" totalInitContainerRuntime="557.444948ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:28.189034547 +0000 UTC m=+224.080096480" watchObservedRunningTime="2026-11-03 23:33:28.197621137 +0000 UTC m=+224.088683065"
Nov 03 23:33:34.829425 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:33:34.829365    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:33:35.076136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.076101    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"]
Nov 03 23:33:35.079217 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.079199    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:35.094476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.094447    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 23:33:35.094604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.094481    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 23:33:35.094604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.094498    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7475g\""
Nov 03 23:33:35.094604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.094482    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 23:33:35.094772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.094755    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"]
Nov 03 23:33:35.265135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.265091    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkm5\" (UniqueName: \"kubernetes.io/projected/9633fbd0-0587-402d-bdd0-ea2072eaeee5-kube-api-access-tqkm5\") pod \"dns-operator-controller-manager-8557cbdb84-9szhc\" (UID: \"9633fbd0-0587-402d-bdd0-ea2072eaeee5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:35.366155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.366041    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkm5\" (UniqueName: \"kubernetes.io/projected/9633fbd0-0587-402d-bdd0-ea2072eaeee5-kube-api-access-tqkm5\") pod \"dns-operator-controller-manager-8557cbdb84-9szhc\" (UID: \"9633fbd0-0587-402d-bdd0-ea2072eaeee5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:35.387197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.387166    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkm5\" (UniqueName: \"kubernetes.io/projected/9633fbd0-0587-402d-bdd0-ea2072eaeee5-kube-api-access-tqkm5\") pod \"dns-operator-controller-manager-8557cbdb84-9szhc\" (UID: \"9633fbd0-0587-402d-bdd0-ea2072eaeee5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:35.391991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.391968    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:35.677504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:35.677432    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"]
Nov 03 23:33:35.680825 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:33:35.680795    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9633fbd0_0587_402d_bdd0_ea2072eaeee5.slice/crio-b2bc854d9cd5e3a3fd96e10ee0cf25c660fc751cd6e7484414be46b1d76fd4a0 WatchSource:0}: Error finding container b2bc854d9cd5e3a3fd96e10ee0cf25c660fc751cd6e7484414be46b1d76fd4a0: Status 404 returned error can't find the container with id b2bc854d9cd5e3a3fd96e10ee0cf25c660fc751cd6e7484414be46b1d76fd4a0
Nov 03 23:33:36.118758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:36.118722    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc" event={"ID":"9633fbd0-0587-402d-bdd0-ea2072eaeee5","Type":"ContainerStarted","Data":"b2bc854d9cd5e3a3fd96e10ee0cf25c660fc751cd6e7484414be46b1d76fd4a0"}
Nov 03 23:33:38.071461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.071425    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"]
Nov 03 23:33:38.075019 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.074992    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:38.090415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.087367    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-prq7d\""
Nov 03 23:33:38.095520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.095494    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"]
Nov 03 23:33:38.194329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.194292    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tpv\" (UniqueName: \"kubernetes.io/projected/c8a4965f-af3d-4332-a17a-92947cef5216-kube-api-access-s4tpv\") pod \"limitador-operator-controller-manager-5778bb4449-rhgs4\" (UID: \"c8a4965f-af3d-4332-a17a-92947cef5216\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:38.295594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.295554    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tpv\" (UniqueName: \"kubernetes.io/projected/c8a4965f-af3d-4332-a17a-92947cef5216-kube-api-access-s4tpv\") pod \"limitador-operator-controller-manager-5778bb4449-rhgs4\" (UID: \"c8a4965f-af3d-4332-a17a-92947cef5216\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:38.324853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.324777    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tpv\" (UniqueName: \"kubernetes.io/projected/c8a4965f-af3d-4332-a17a-92947cef5216-kube-api-access-s4tpv\") pod \"limitador-operator-controller-manager-5778bb4449-rhgs4\" (UID: \"c8a4965f-af3d-4332-a17a-92947cef5216\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:38.397221 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.397178    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:38.707845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:38.707766    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"]
Nov 03 23:33:38.711170 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:33:38.711139    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a4965f_af3d_4332_a17a_92947cef5216.slice/crio-622378839c0039b52660326bd24500623e8adb102f92367eed93b43d15cee09c WatchSource:0}: Error finding container 622378839c0039b52660326bd24500623e8adb102f92367eed93b43d15cee09c: Status 404 returned error can't find the container with id 622378839c0039b52660326bd24500623e8adb102f92367eed93b43d15cee09c
Nov 03 23:33:39.096103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:39.096071    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 03 23:33:39.134287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:39.134249    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc" event={"ID":"9633fbd0-0587-402d-bdd0-ea2072eaeee5","Type":"ContainerStarted","Data":"7f2516943400b4f6a5ee9929786d4d70729718a3695dc16224c16a481b039580"}
Nov 03 23:33:39.134494 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:39.134341    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:39.135609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:39.135580    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerStarted","Data":"622378839c0039b52660326bd24500623e8adb102f92367eed93b43d15cee09c"}
Nov 03 23:33:39.217186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:39.217129    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc" podStartSLOduration=1.4662925310000001 podStartE2EDuration="4.217111373s" totalImagesPullingTime="2.750818842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:39.215187517 +0000 UTC m=+235.106249451" watchObservedRunningTime="2026-11-03 23:33:39.217111373 +0000 UTC m=+235.108173297"
Nov 03 23:33:42.149035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:42.148999    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerStarted","Data":"ba8726203bf2f40165e7e2d72d284cf1a475c0ef70e08ed62532570e4cd7ed3f"}
Nov 03 23:33:42.149463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:42.149104    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:42.217304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:42.217252    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" podStartSLOduration=1.404545875 podStartE2EDuration="4.217235429s" totalImagesPullingTime="2.812689554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:42.214304152 +0000 UTC m=+238.105366085" watchObservedRunningTime="2026-11-03 23:33:42.217235429 +0000 UTC m=+238.108297363"
Nov 03 23:33:50.142586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:50.142549    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9szhc"
Nov 03 23:33:53.155453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:53.155423    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 03 23:33:57.433240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.433205    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"]
Nov 03 23:33:57.439873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.439849    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.453853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.453788    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"]
Nov 03 23:33:57.507982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.507948    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 23:33:57.508151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.508088    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 23:33:57.508529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.508510    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pc7rt\""
Nov 03 23:33:57.572153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.572110    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2354e993-f4ef-4b87-956b-b35a01cef507-tls-certs\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.572350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.572165    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdg5\" (UniqueName: \"kubernetes.io/projected/2354e993-f4ef-4b87-956b-b35a01cef507-kube-api-access-5zdg5\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.572350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.572300    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2354e993-f4ef-4b87-956b-b35a01cef507-tmp\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.673762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.673724    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2354e993-f4ef-4b87-956b-b35a01cef507-tmp\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.673960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.673789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2354e993-f4ef-4b87-956b-b35a01cef507-tls-certs\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.673960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.673822    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdg5\" (UniqueName: \"kubernetes.io/projected/2354e993-f4ef-4b87-956b-b35a01cef507-kube-api-access-5zdg5\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.676041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.676014    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2354e993-f4ef-4b87-956b-b35a01cef507-tmp\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.676272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.676248    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2354e993-f4ef-4b87-956b-b35a01cef507-tls-certs\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.697863 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.697802    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdg5\" (UniqueName: \"kubernetes.io/projected/2354e993-f4ef-4b87-956b-b35a01cef507-kube-api-access-5zdg5\") pod \"kube-auth-proxy-694bd9f98-bqzd8\" (UID: \"2354e993-f4ef-4b87-956b-b35a01cef507\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:57.753298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:57.753258    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"
Nov 03 23:33:58.031149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:58.031065    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8"]
Nov 03 23:33:58.037315 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:33:58.037284    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2354e993_f4ef_4b87_956b_b35a01cef507.slice/crio-5e4aff4e6d3e033a8e802822a4505d8135f2bbd2237e942583b716da17a4720e WatchSource:0}: Error finding container 5e4aff4e6d3e033a8e802822a4505d8135f2bbd2237e942583b716da17a4720e: Status 404 returned error can't find the container with id 5e4aff4e6d3e033a8e802822a4505d8135f2bbd2237e942583b716da17a4720e
Nov 03 23:33:58.211413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:33:58.211366    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8" event={"ID":"2354e993-f4ef-4b87-956b-b35a01cef507","Type":"ContainerStarted","Data":"5e4aff4e6d3e033a8e802822a4505d8135f2bbd2237e942583b716da17a4720e"}
Nov 03 23:34:01.226449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:01.226344    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8" event={"ID":"2354e993-f4ef-4b87-956b-b35a01cef507","Type":"ContainerStarted","Data":"14b3fd07b03ad2f66673935b595328441eb1991c7510a42026cf909607da9b46"}
Nov 03 23:34:01.272118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:01.272051    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-694bd9f98-bqzd8" podStartSLOduration=1.362936924 podStartE2EDuration="4.272031702s" totalImagesPullingTime="2.909094778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:01.269799253 +0000 UTC m=+257.160861189" watchObservedRunningTime="2026-11-03 23:34:01.272031702 +0000 UTC m=+257.163093640"
Nov 03 23:34:04.829552 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:34:04.829514    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:34:07.902894 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.902856    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:34:07.905296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.905277    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:07.918857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.918838    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 23:34:07.918956 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.918883    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pkmb4\""
Nov 03 23:34:07.921047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.921023    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:34:07.952375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:07.952346    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:34:08.069140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.069095    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f95bcd-d2d7-481a-b499-3458acd41ddd-config-file\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.069316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.069156    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ckz\" (UniqueName: \"kubernetes.io/projected/84f95bcd-d2d7-481a-b499-3458acd41ddd-kube-api-access-42ckz\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.170295 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.170199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f95bcd-d2d7-481a-b499-3458acd41ddd-config-file\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.170295 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.170253    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ckz\" (UniqueName: \"kubernetes.io/projected/84f95bcd-d2d7-481a-b499-3458acd41ddd-kube-api-access-42ckz\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.171046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.171019    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f95bcd-d2d7-481a-b499-3458acd41ddd-config-file\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.185036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.185002    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ckz\" (UniqueName: \"kubernetes.io/projected/84f95bcd-d2d7-481a-b499-3458acd41ddd-kube-api-access-42ckz\") pod \"limitador-limitador-7c9b87d574-jt564\" (UID: \"84f95bcd-d2d7-481a-b499-3458acd41ddd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.216860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.216818    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:08.495075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:08.494987    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:34:08.498046 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:34:08.498006    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f95bcd_d2d7_481a_b499_3458acd41ddd.slice/crio-3770549f854ed83bd2449a8de595fa79cb30e3ddd7b89deb3275e462b62a7daa WatchSource:0}: Error finding container 3770549f854ed83bd2449a8de595fa79cb30e3ddd7b89deb3275e462b62a7daa: Status 404 returned error can't find the container with id 3770549f854ed83bd2449a8de595fa79cb30e3ddd7b89deb3275e462b62a7daa
Nov 03 23:34:09.256482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:09.256442    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564" event={"ID":"84f95bcd-d2d7-481a-b499-3458acd41ddd","Type":"ContainerStarted","Data":"3770549f854ed83bd2449a8de595fa79cb30e3ddd7b89deb3275e462b62a7daa"}
Nov 03 23:34:10.261459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:10.261354    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564" event={"ID":"84f95bcd-d2d7-481a-b499-3458acd41ddd","Type":"ContainerStarted","Data":"43a0595bff2b42038988c5e32418a1ad650d2b0a2640ef018fc52b41a9548cd2"}
Nov 03 23:34:10.261833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:10.261480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:10.283808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:10.283751    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564" podStartSLOduration=1.959890379 podStartE2EDuration="3.283737541s" totalImagesPullingTime="1.323847162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:10.283008785 +0000 UTC m=+266.174070719" watchObservedRunningTime="2026-11-03 23:34:10.283737541 +0000 UTC m=+266.174799470"
Nov 03 23:34:21.267067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:21.267032    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jt564"
Nov 03 23:34:34.829901 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:34:34.829864    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:34:55.642705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.642671    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 03 23:34:55.645990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.645968    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.651590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.651566    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 23:34:55.651590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.651585    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 23:34:55.657234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.657211    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 03 23:34:55.658777 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.658753    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 23:34:55.658885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.658809    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nt628\""
Nov 03 23:34:55.658885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.658822    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 23:34:55.757156 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.757120    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.757325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.757199    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.757325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.757237    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnmn\" (UniqueName: \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.856560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.856531    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"]
Nov 03 23:34:55.857962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.857931    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.858090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.858002    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnmn\" (UniqueName: \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.858141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.858092    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.859870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.859847    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.860619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.860593    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.860715 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.860694    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.871908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.871862    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnmn\" (UniqueName: \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\") pod \"odh-model-controller-769875db56-n76xc\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") " pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.877424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.877380    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"]
Nov 03 23:34:55.958842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.958756    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.958842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.958793    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.958842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.958814    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.959057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.958918    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.959057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.959008    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.959057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.959039    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:55.959162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.959056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:55.959162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:55.959087    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zltm\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-kube-api-access-7zltm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060052    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zltm\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-kube-api-access-7zltm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060112    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060137    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060160    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060200    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.060568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.060292    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.061028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.061001    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.062710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.062681    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.062801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.062707    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.062844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.062833    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.062959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.062930    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.073461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.073425    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.073573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.073538    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zltm\" (UniqueName: \"kubernetes.io/projected/f7abbbc7-1072-4563-9ebd-1ebe917ebca5-kube-api-access-7zltm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hhzdd\" (UID: \"f7abbbc7-1072-4563-9ebd-1ebe917ebca5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.192814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.192783    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:56.265277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.265249    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 03 23:34:56.275879 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:34:56.275853    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099113e5_3a77_4970_9aeb_2ce66f6de53c.slice/crio-99fef0ea846cc57ead56a6e67e308e5be1b497bac14099c648067523d4d68b93 WatchSource:0}: Error finding container 99fef0ea846cc57ead56a6e67e308e5be1b497bac14099c648067523d4d68b93: Status 404 returned error can't find the container with id 99fef0ea846cc57ead56a6e67e308e5be1b497bac14099c648067523d4d68b93
Nov 03 23:34:56.277067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.277050    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:34:56.444798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.444761    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n76xc" event={"ID":"099113e5-3a77-4970-9aeb-2ce66f6de53c","Type":"ContainerStarted","Data":"99fef0ea846cc57ead56a6e67e308e5be1b497bac14099c648067523d4d68b93"}
Nov 03 23:34:56.470999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.470962    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"]
Nov 03 23:34:56.473900 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:34:56.473871    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7abbbc7_1072_4563_9ebd_1ebe917ebca5.slice/crio-c8801ad8475891a354a1d9df922263fd4fc1cd0ae93a0ac740a6ebb8b884ce5f WatchSource:0}: Error finding container c8801ad8475891a354a1d9df922263fd4fc1cd0ae93a0ac740a6ebb8b884ce5f: Status 404 returned error can't find the container with id c8801ad8475891a354a1d9df922263fd4fc1cd0ae93a0ac740a6ebb8b884ce5f
Nov 03 23:34:56.475880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.475850    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:34:56.475954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:56.475908    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:34:57.452614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.452566    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd" event={"ID":"f7abbbc7-1072-4563-9ebd-1ebe917ebca5","Type":"ContainerStarted","Data":"0aa8ccc679bd897591950d5fee7572ae18b027798cd84dbf4e8fc8cc4f906209"}
Nov 03 23:34:57.453056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.452637    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd" event={"ID":"f7abbbc7-1072-4563-9ebd-1ebe917ebca5","Type":"ContainerStarted","Data":"c8801ad8475891a354a1d9df922263fd4fc1cd0ae93a0ac740a6ebb8b884ce5f"}
Nov 03 23:34:57.453056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.452778    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:57.454833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.454811    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd"
Nov 03 23:34:57.483667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.482557    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hhzdd" podStartSLOduration=2.482541492 podStartE2EDuration="2.482541492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:57.482356229 +0000 UTC m=+313.373418176" watchObservedRunningTime="2026-11-03 23:34:57.482541492 +0000 UTC m=+313.373603426"
Nov 03 23:34:57.543874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.543828    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:34:57.544108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.544066    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" podUID="f99b3523-8e07-4982-bb6f-35adaec7cf68" containerName="discovery" containerID="cri-o://a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a" gracePeriod=30
Nov 03 23:34:57.831863 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.831829    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:34:57.984239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984162    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984231    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984319    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984359    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984418    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984443    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984462    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\") pod \"f99b3523-8e07-4982-bb6f-35adaec7cf68\" (UID: \"f99b3523-8e07-4982-bb6f-35adaec7cf68\") "
Nov 03 23:34:57.984750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984606    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:34:57.984750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.984707    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-ca-configmap\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:57.989571 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.989520    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:34:57.989873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.989837    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:34:57.989873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.989849    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "kube-api-access-7rnzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:34:57.990010 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.989949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:34:57.990010 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.989949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:34:57.990277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:57.990256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig" pod "f99b3523-8e07-4982-bb6f-35adaec7cf68" (UID: "f99b3523-8e07-4982-bb6f-35adaec7cf68"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:34:58.085817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085777    6430 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-csr-dns-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.085817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085814    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnzm\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-kube-api-access-7rnzm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.085817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085825    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.086071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085834    6430 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-istio-kubeconfig\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.086071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085843    6430 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99b3523-8e07-4982-bb6f-35adaec7cf68-local-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.086071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.085850    6430 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f99b3523-8e07-4982-bb6f-35adaec7cf68-cacerts\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:34:58.458682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.458649    6430 generic.go:356] "Generic (PLEG): container finished" podID="f99b3523-8e07-4982-bb6f-35adaec7cf68" containerID="a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a" exitCode=0
Nov 03 23:34:58.459141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.458714    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"
Nov 03 23:34:58.459141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.458744    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" event={"ID":"f99b3523-8e07-4982-bb6f-35adaec7cf68","Type":"ContainerDied","Data":"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"}
Nov 03 23:34:58.459141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.458809    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k" event={"ID":"f99b3523-8e07-4982-bb6f-35adaec7cf68","Type":"ContainerDied","Data":"f73f125916f558fc72461497a4ccf18abe02c591f7f8be037b381efd34ae485b"}
Nov 03 23:34:58.459141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.458842    6430 scope.go:122] "RemoveContainer" containerID="a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"
Nov 03 23:34:58.467850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.467821    6430 scope.go:122] "RemoveContainer" containerID="a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"
Nov 03 23:34:58.468161 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:34:58.468135    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a\": container with ID starting with a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a not found: ID does not exist" containerID="a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"
Nov 03 23:34:58.468261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.468173    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a"} err="failed to get container status \"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a\": rpc error: code = NotFound desc = could not find container \"a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a\": container with ID starting with a10803e962346353476547c9339373bf91b4b11fca11ec65cd1f84733042140a not found: ID does not exist"
Nov 03 23:34:58.481275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.481249    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:34:58.485645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:58.485621    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5t59k"]
Nov 03 23:34:59.464045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:59.464009    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n76xc" event={"ID":"099113e5-3a77-4970-9aeb-2ce66f6de53c","Type":"ContainerStarted","Data":"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"}
Nov 03 23:34:59.464552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:59.464208    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:34:59.485378 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:34:59.485323    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-n76xc" podStartSLOduration=1.746825856 podStartE2EDuration="4.485310849s" totalImagesPullingTime="2.738484993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:59.484302863 +0000 UTC m=+315.375364799" watchObservedRunningTime="2026-11-03 23:34:59.485310849 +0000 UTC m=+315.376373257"
Nov 03 23:35:00.229433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:00.229382    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99b3523-8e07-4982-bb6f-35adaec7cf68" path="/var/lib/kubelet/pods/f99b3523-8e07-4982-bb6f-35adaec7cf68/volumes"
Nov 03 23:35:04.831494 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:35:04.831448    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:35:10.472092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:10.472055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 03 23:35:31.898922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:31.898882    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:35:34.830337 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:35:34.830297    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:35:43.996686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:43.996650    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 03 23:35:43.997148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:43.996991    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f99b3523-8e07-4982-bb6f-35adaec7cf68" containerName="discovery"
Nov 03 23:35:43.997148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:43.997003    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99b3523-8e07-4982-bb6f-35adaec7cf68" containerName="discovery"
Nov 03 23:35:43.997148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:43.997082    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99b3523-8e07-4982-bb6f-35adaec7cf68" containerName="discovery"
Nov 03 23:35:44.000274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.000252    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.010224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.010196    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 03 23:35:44.035555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.035527    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 23:35:44.035678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.035621    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 23:35:44.048080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.048059    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\""
Nov 03 23:35:44.048137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.048128    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 23:35:44.048199 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.048189    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 23:35:44.079417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.079370    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6mv\" (UniqueName: \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.079567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.079446    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.079567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.079509    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.079679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.079578    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.180858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.180821    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.180858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.180862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.181116 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.180928    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6mv\" (UniqueName: \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.181116 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.180980    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.185088 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.185061    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 23:35:44.185223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.185085    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 23:35:44.185223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.185094    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 23:35:44.191804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.191779    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.193584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.193562    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.193670 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.193586    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.197996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.197977    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 23:35:44.206795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.206774    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6mv\" (UniqueName: \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\") pod \"maas-api-59f797486c-g9klm\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.317364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.317328    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\""
Nov 03 23:35:44.323996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.323971    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:44.600567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.600545    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 03 23:35:44.602974 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:35:44.602946    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59db20c6_b9de_4de9_b16d_48a648fc7e8b.slice/crio-bce86a196216d181557a436c38041384e95205022283967c502b690590992d3f WatchSource:0}: Error finding container bce86a196216d181557a436c38041384e95205022283967c502b690590992d3f: Status 404 returned error can't find the container with id bce86a196216d181557a436c38041384e95205022283967c502b690590992d3f
Nov 03 23:35:44.661441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:44.661406    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" event={"ID":"59db20c6-b9de-4de9-b16d-48a648fc7e8b","Type":"ContainerStarted","Data":"bce86a196216d181557a436c38041384e95205022283967c502b690590992d3f"}
Nov 03 23:35:47.677248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:47.677210    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" event={"ID":"59db20c6-b9de-4de9-b16d-48a648fc7e8b","Type":"ContainerStarted","Data":"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"}
Nov 03 23:35:47.677661 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:47.677287    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:35:47.701548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:47.701498    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" podStartSLOduration=2.332747174 podStartE2EDuration="4.701484581s" totalImagesPullingTime="2.368737407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:35:47.700510147 +0000 UTC m=+363.591572083" watchObservedRunningTime="2026-11-03 23:35:47.701484581 +0000 UTC m=+363.592546515"
Nov 03 23:35:53.685862 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:35:53.685786    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 03 23:36:04.829432 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:36:04.829375    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:36:34.829801 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:36:34.829762    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:37:04.829712 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:37:04.829675    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:37:34.831139 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:37:34.831101    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:37:35.403168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.403131    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"]
Nov 03 23:37:35.406597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.406571    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.413215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.413089    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vfcjl\""
Nov 03 23:37:35.417068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.417036    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 23:37:35.418871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.418843    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 23:37:35.419015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.418994    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 23:37:35.419947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.419921    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"]
Nov 03 23:37:35.453850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.453815    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcab315-9f9c-4214-9fd7-36811b39c65b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.453850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.453854    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.454058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.453874    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvsq\" (UniqueName: \"kubernetes.io/projected/bfcab315-9f9c-4214-9fd7-36811b39c65b-kube-api-access-dwvsq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.454058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.453983    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.454058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.454036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.454167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.454101    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.554977    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcab315-9f9c-4214-9fd7-36811b39c65b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555021    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555154    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvsq\" (UniqueName: \"kubernetes.io/projected/bfcab315-9f9c-4214-9fd7-36811b39c65b-kube-api-access-dwvsq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555207    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555232    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555308    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555467    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.555802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.555779    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.557491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.557464    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfcab315-9f9c-4214-9fd7-36811b39c65b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.557755 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.557735    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfcab315-9f9c-4214-9fd7-36811b39c65b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.569090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.569061    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvsq\" (UniqueName: \"kubernetes.io/projected/bfcab315-9f9c-4214-9fd7-36811b39c65b-kube-api-access-dwvsq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh\" (UID: \"bfcab315-9f9c-4214-9fd7-36811b39c65b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.721180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.721080    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:35.929291 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:35.929255    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:37:36.000756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:36.000678    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"]
Nov 03 23:37:36.004411 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:36.004372    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcab315_9f9c_4214_9fd7_36811b39c65b.slice/crio-b8a398b651e5081784e049971513d5f8c990185eaf8f3a4d72d1a8aecdb45319 WatchSource:0}: Error finding container b8a398b651e5081784e049971513d5f8c990185eaf8f3a4d72d1a8aecdb45319: Status 404 returned error can't find the container with id b8a398b651e5081784e049971513d5f8c990185eaf8f3a4d72d1a8aecdb45319
Nov 03 23:37:36.118245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:36.118207    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" event={"ID":"bfcab315-9f9c-4214-9fd7-36811b39c65b","Type":"ContainerStarted","Data":"b8a398b651e5081784e049971513d5f8c990185eaf8f3a4d72d1a8aecdb45319"}
Nov 03 23:37:38.128436 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.128397    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" event={"ID":"bfcab315-9f9c-4214-9fd7-36811b39c65b","Type":"ContainerStarted","Data":"7bb4e3e5edf2bff74a68d98357cf95575c2a2178acbd889bea96f874dc74750e"}
Nov 03 23:37:38.149784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.149729    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" podStartSLOduration=1.761556289 podStartE2EDuration="3.149712998s" totalImagesPullingTime="1.388156709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:38.148990288 +0000 UTC m=+474.040052226" watchObservedRunningTime="2026-11-03 23:37:38.149712998 +0000 UTC m=+474.040774927"
Nov 03 23:37:38.639548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.639508    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:37:38.714807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.714777    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"]
Nov 03 23:37:38.718175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.718155    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.721886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.721865    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:38.724991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.724971    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 23:37:38.729926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.729902    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"]
Nov 03 23:37:38.787091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787052    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.787310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787117    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx6t\" (UniqueName: \"kubernetes.io/projected/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-kube-api-access-2tx6t\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.787310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787202    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.787310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787293    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.787532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787367    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.787532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.787416    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx6t\" (UniqueName: \"kubernetes.io/projected/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-kube-api-access-2tx6t\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888423    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888481    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888511    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888573    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888593    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888821    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.888912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.888844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.890785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.890726    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.890923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.890905    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:38.932400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:38.932359    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx6t\" (UniqueName: \"kubernetes.io/projected/9a6ecec9-7825-4edb-9cb2-4a38bca0a52c-kube-api-access-2tx6t\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w2wvx\" (UID: \"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:39.031334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:39.031284    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:39.306294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:39.306266    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"]
Nov 03 23:37:39.308992 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:39.308965    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ecec9_7825_4edb_9cb2_4a38bca0a52c.slice/crio-a189121c0afa1ba63965ab1e305b250750a0341de9fa7c05e5b3824535e243fb WatchSource:0}: Error finding container a189121c0afa1ba63965ab1e305b250750a0341de9fa7c05e5b3824535e243fb: Status 404 returned error can't find the container with id a189121c0afa1ba63965ab1e305b250750a0341de9fa7c05e5b3824535e243fb
Nov 03 23:37:40.137598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.137509    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx" event={"ID":"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c","Type":"ContainerStarted","Data":"7e6af36b4a0ce4da1a442df64278d962a02ba6f8885cf63f73f8a30c8e3579ea"}
Nov 03 23:37:40.137598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.137566    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx" event={"ID":"9a6ecec9-7825-4edb-9cb2-4a38bca0a52c","Type":"ContainerStarted","Data":"a189121c0afa1ba63965ab1e305b250750a0341de9fa7c05e5b3824535e243fb"}
Nov 03 23:37:40.164628 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.164582    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx" podStartSLOduration=1.6411750139999999 podStartE2EDuration="2.164568274s" totalImagesPullingTime="523.39326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:40.161975524 +0000 UTC m=+476.053037478" watchObservedRunningTime="2026-11-03 23:37:40.164568274 +0000 UTC m=+476.055630208"
Nov 03 23:37:40.321684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.321656    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"]
Nov 03 23:37:40.325255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.325235    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.344949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.344923    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 23:37:40.346608 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.346587    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"]
Nov 03 23:37:40.402030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.401953    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6bt\" (UniqueName: \"kubernetes.io/projected/87912bc2-fd56-4bbd-81f1-e31e390f0727-kube-api-access-hd6bt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.402030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.401990    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.402030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.402021    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.402314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.402073    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87912bc2-fd56-4bbd-81f1-e31e390f0727-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.402314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.402233    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.402445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.402314    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.503806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.503776    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6bt\" (UniqueName: \"kubernetes.io/projected/87912bc2-fd56-4bbd-81f1-e31e390f0727-kube-api-access-hd6bt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.503806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.503811    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.503833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.503916    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87912bc2-fd56-4bbd-81f1-e31e390f0727-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.503968    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.504027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.504252    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.504274    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.504382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.504357    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.506233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.506211    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87912bc2-fd56-4bbd-81f1-e31e390f0727-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.506423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.506403    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87912bc2-fd56-4bbd-81f1-e31e390f0727-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.525545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.525513    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6bt\" (UniqueName: \"kubernetes.io/projected/87912bc2-fd56-4bbd-81f1-e31e390f0727-kube-api-access-hd6bt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj\" (UID: \"87912bc2-fd56-4bbd-81f1-e31e390f0727\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.638867 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.638834    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:40.923953 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:40.923921    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"]
Nov 03 23:37:40.927081 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:40.927055    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87912bc2_fd56_4bbd_81f1_e31e390f0727.slice/crio-03bff79a9d998e40fd1913e2ee009d2987798e29fa44414fd259934a609c53ad WatchSource:0}: Error finding container 03bff79a9d998e40fd1913e2ee009d2987798e29fa44414fd259934a609c53ad: Status 404 returned error can't find the container with id 03bff79a9d998e40fd1913e2ee009d2987798e29fa44414fd259934a609c53ad
Nov 03 23:37:41.031800 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:41.031715    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:41.142576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:41.142542    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj" event={"ID":"87912bc2-fd56-4bbd-81f1-e31e390f0727","Type":"ContainerStarted","Data":"03bff79a9d998e40fd1913e2ee009d2987798e29fa44414fd259934a609c53ad"}
Nov 03 23:37:41.196236 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:41.196199    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:37:42.147905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.147866    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj" event={"ID":"87912bc2-fd56-4bbd-81f1-e31e390f0727","Type":"ContainerStarted","Data":"703e7bc9d541b5c3dab6ed5755890c579656c41ef1422c5c0e9b499e55cfd524"}
Nov 03 23:37:42.181492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.181445    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj" podStartSLOduration=1.900182424 podStartE2EDuration="2.18143138s" totalImagesPullingTime="281.248956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:42.181238051 +0000 UTC m=+478.072299989" watchObservedRunningTime="2026-11-03 23:37:42.18143138 +0000 UTC m=+478.072493313"
Nov 03 23:37:42.639327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.639292    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:42.947248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.947165    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"]
Nov 03 23:37:42.951060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.951034    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:42.961527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.961501    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 23:37:42.963298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:42.963255    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"]
Nov 03 23:37:43.028401 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028360    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff13fca-8c3c-483a-92aa-b8844a6712af-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.028565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028425    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.028565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028544    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.028648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028608    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.028681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028653    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.028717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.028703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6dm\" (UniqueName: \"kubernetes.io/projected/eff13fca-8c3c-483a-92aa-b8844a6712af-kube-api-access-jq6dm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130145    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130206    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130278    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6dm\" (UniqueName: \"kubernetes.io/projected/eff13fca-8c3c-483a-92aa-b8844a6712af-kube-api-access-jq6dm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130318    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff13fca-8c3c-483a-92aa-b8844a6712af-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130351    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130375    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130671 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130674    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.130793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.130734    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.132627 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.132600    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff13fca-8c3c-483a-92aa-b8844a6712af-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.132783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.132768    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff13fca-8c3c-483a-92aa-b8844a6712af-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.147487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.147464    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6dm\" (UniqueName: \"kubernetes.io/projected/eff13fca-8c3c-483a-92aa-b8844a6712af-kube-api-access-jq6dm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd\" (UID: \"eff13fca-8c3c-483a-92aa-b8844a6712af\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.263861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.263772    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:43.542947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:43.542907    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"]
Nov 03 23:37:43.546247 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:43.546213    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff13fca_8c3c_483a_92aa_b8844a6712af.slice/crio-3575a5b15e7c90630b8f298842ec68d301f17bba5f93aae34bced44bee772b26 WatchSource:0}: Error finding container 3575a5b15e7c90630b8f298842ec68d301f17bba5f93aae34bced44bee772b26: Status 404 returned error can't find the container with id 3575a5b15e7c90630b8f298842ec68d301f17bba5f93aae34bced44bee772b26
Nov 03 23:37:44.157526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.157432    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd" event={"ID":"eff13fca-8c3c-483a-92aa-b8844a6712af","Type":"ContainerStarted","Data":"b6eb603611f8c15c61678814abe4031ea5f793e527ed1177c933bd917acd3c55"}
Nov 03 23:37:44.157526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.157497    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd" event={"ID":"eff13fca-8c3c-483a-92aa-b8844a6712af","Type":"ContainerStarted","Data":"3575a5b15e7c90630b8f298842ec68d301f17bba5f93aae34bced44bee772b26"}
Nov 03 23:37:44.181909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.181856    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd" podStartSLOduration=1.884194368 podStartE2EDuration="2.181842798s" totalImagesPullingTime="297.64843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:44.181597929 +0000 UTC m=+480.072659862" watchObservedRunningTime="2026-11-03 23:37:44.181842798 +0000 UTC m=+480.072904731"
Nov 03 23:37:44.264558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.264530    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:44.326579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.324710    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"]
Nov 03 23:37:44.329156 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.329124    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.336081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.336047    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 23:37:44.341974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.341953    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"]
Nov 03 23:37:44.442588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442501    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.442588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442540    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.442588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442568    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.442806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21599918-1626-411b-8922-0270ebf1e061-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.442806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442734    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.442806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.442764    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2zm\" (UniqueName: \"kubernetes.io/projected/21599918-1626-411b-8922-0270ebf1e061-kube-api-access-dq2zm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.543970    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544046    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21599918-1626-411b-8922-0270ebf1e061-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544105    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2zm\" (UniqueName: \"kubernetes.io/projected/21599918-1626-411b-8922-0270ebf1e061-kube-api-access-dq2zm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544192    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544218    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544411    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544469    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.544626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.544602    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.546503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.546475    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21599918-1626-411b-8922-0270ebf1e061-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.546672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.546657    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21599918-1626-411b-8922-0270ebf1e061-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.557270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.557233    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2zm\" (UniqueName: \"kubernetes.io/projected/21599918-1626-411b-8922-0270ebf1e061-kube-api-access-dq2zm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd\" (UID: \"21599918-1626-411b-8922-0270ebf1e061\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.655999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.655962    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:44.930915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:44.930884    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"]
Nov 03 23:37:44.934079 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:44.934050    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21599918_1626_411b_8922_0270ebf1e061.slice/crio-cf0c18b342abe53b27271a41f9ebfc73dbd97df3ebb2b8cf1c5e5058bbe90e5d WatchSource:0}: Error finding container cf0c18b342abe53b27271a41f9ebfc73dbd97df3ebb2b8cf1c5e5058bbe90e5d: Status 404 returned error can't find the container with id cf0c18b342abe53b27271a41f9ebfc73dbd97df3ebb2b8cf1c5e5058bbe90e5d
Nov 03 23:37:45.162465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:45.162424    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" event={"ID":"21599918-1626-411b-8922-0270ebf1e061","Type":"ContainerStarted","Data":"cf0c18b342abe53b27271a41f9ebfc73dbd97df3ebb2b8cf1c5e5058bbe90e5d"}
Nov 03 23:37:45.722182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:45.722143    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:45.735059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:45.735029    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:46.174565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:46.174525    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" event={"ID":"21599918-1626-411b-8922-0270ebf1e061","Type":"ContainerStarted","Data":"cd54f9905ccbc00f08163233f5f278d78da429fae17a0ee7d9a5a925ea05d951"}
Nov 03 23:37:46.186057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:46.186029    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh"
Nov 03 23:37:46.197111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:46.197062    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podStartSLOduration=1.881283908 podStartE2EDuration="2.197049697s" totalImagesPullingTime="315.765789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:46.196424599 +0000 UTC m=+482.087486536" watchObservedRunningTime="2026-11-03 23:37:46.197049697 +0000 UTC m=+482.088111626"
Nov 03 23:37:46.657160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:46.657121    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:46.876660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:46.876623    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:37:47.282413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.282360    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 03 23:37:47.288261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.288238    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.304877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.304845    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 03 23:37:47.370894 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.370857    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.370894 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.370895    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.370922    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf82h\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371074    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371117    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371230    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371264    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371284    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.371311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.371300    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.384636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.384608    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sh6t7\""
Nov 03 23:37:47.472243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472200    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472306    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472339    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472365    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472414    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472456    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472479    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472512    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf82h\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472572    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472749    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.472996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.472959    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.473104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.473082    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.473104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.473087    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.473277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.473174    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.474654 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.474630    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.474904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.474885    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.505039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.504999    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.505183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.505117    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf82h\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\") pod \"maas-default-gateway-openshift-default-778b7b5c47-n9b2x\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.602721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.602683    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:47.885682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:47.885657    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 03 23:37:47.888378 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:37:47.888347    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857b77fc_a180_4a57_b444_977daca6a832.slice/crio-b60a58d500f12fa9c857dfe93f2218fd93140a179d82ac47363a18130f09e2ad WatchSource:0}: Error finding container b60a58d500f12fa9c857dfe93f2218fd93140a179d82ac47363a18130f09e2ad: Status 404 returned error can't find the container with id b60a58d500f12fa9c857dfe93f2218fd93140a179d82ac47363a18130f09e2ad
Nov 03 23:37:48.186997 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:48.186902    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" event={"ID":"857b77fc-a180-4a57-b444-977daca6a832","Type":"ContainerStarted","Data":"b60a58d500f12fa9c857dfe93f2218fd93140a179d82ac47363a18130f09e2ad"}
Nov 03 23:37:49.032354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:49.032317    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:49.045627 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:49.045596    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:49.206173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:49.206141    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w2wvx"
Nov 03 23:37:50.639881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:50.639842    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:50.654122 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:50.654090    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:50.947813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:50.947780    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:37:50.947917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:50.947867    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:37:50.947917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:50.947902    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:37:51.203165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:51.203067    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" event={"ID":"857b77fc-a180-4a57-b444-977daca6a832","Type":"ContainerStarted","Data":"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"}
Nov 03 23:37:51.214886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:51.214859    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj"
Nov 03 23:37:51.229263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:51.229215    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podStartSLOduration=1.172704523 podStartE2EDuration="4.229202091s" totalImagesPullingTime="3.056497568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:51.228214578 +0000 UTC m=+487.119276519" watchObservedRunningTime="2026-11-03 23:37:51.229202091 +0000 UTC m=+487.120264025"
Nov 03 23:37:51.603446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:51.603411    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:52.605085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:52.605043    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:52.605500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:52.605103    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:53.264947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:53.264907    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:53.278343 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:53.278315    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:53.603766 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:53.603727    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:53.603952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:53.603796    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:54.250835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:54.250807    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd"
Nov 03 23:37:54.603457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:54.603418    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:54.603647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:54.603486    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:54.656466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:54.656430    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:54.669134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:54.669103    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:55.255076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:55.255047    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:37:55.603092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:55.603053    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:55.603273 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:55.603112    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:56.604173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:56.604133    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:56.604611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:56.604195    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:57.604103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:57.604062    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 23:37:57.604307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:57.604121    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 23:37:59.351272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:59.350727    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:59.351780 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:59.351446    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:37:59.352233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:37:59.352213    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 03 23:38:02.338897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.338849    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:38:02.345587 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.345557    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.363554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.363518    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:38:02.438408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438357    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438477    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438595    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438618    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438636    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438774    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438832 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438817    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.438886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.438872    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79jj\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540150 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540112    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540198    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79jj\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540260    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540291    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540324    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540495    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540541    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540763    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.540986 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.540959    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.541051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.541028    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.541089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.541042    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.541122 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.541041    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.542649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.542623    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.543175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.543155    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.553521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.553494    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.554057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.554039    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79jj\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-m8wgk\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.660383 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.660296    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:02.944773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.944746    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:38:02.947345 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:38:02.947318    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc2b3c6_e436_48c9_ba56_19747ae764bc.slice/crio-77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45 WatchSource:0}: Error finding container 77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45: Status 404 returned error can't find the container with id 77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45
Nov 03 23:38:02.950474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.950444    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:02.950548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.950512    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:02.950585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:02.950545    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:03.282202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:03.282103    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" event={"ID":"5bc2b3c6-e436-48c9-ba56-19747ae764bc","Type":"ContainerStarted","Data":"7cc91982ec84a68c692b081b7784cad19f672878aac2501bb3adfe7b10c10b4f"}
Nov 03 23:38:03.282202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:03.282173    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" event={"ID":"5bc2b3c6-e436-48c9-ba56-19747ae764bc","Type":"ContainerStarted","Data":"77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45"}
Nov 03 23:38:03.312213 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:03.312160    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podStartSLOduration=1.312145079 podStartE2EDuration="1.312145079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:03.309417768 +0000 UTC m=+499.200479703" watchObservedRunningTime="2026-11-03 23:38:03.312145079 +0000 UTC m=+499.203207014"
Nov 03 23:38:03.661256 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:03.661211    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:04.662287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:04.662249    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:04.662717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:04.662331    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:04.855660 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:38:04.855609    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:38:05.661021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:05.660978    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:05.661217 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:05.661057    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:06.661641 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:06.661605    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:06.662057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:06.661666    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:07.661098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:07.661057    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:07.661321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:07.661140    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:08.661515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:08.661475    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:08.661927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:08.661532    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:09.661663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:09.661624    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 23:38:09.662082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:09.661679    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 23:38:11.347054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:11.347018    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:11.348208 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:11.348179    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:11.348720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:11.348703    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:38:17.353584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.353545    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 03 23:38:17.361255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.361226    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.376792 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.376756    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 03 23:38:17.491226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491182    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491338    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxf56\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491419    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491481    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491582    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491608    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491628    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.491696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.491647    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.592725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592685    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.592725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592734    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592762    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxf56\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592784    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592867    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592896    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592921    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.592952    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593369 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.593306    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.593591    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.593659    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.593666    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.593801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.593743    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.595133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.595110    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.595474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.595454    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.606710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.606643    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.606796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.606768    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxf56\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\") pod \"maas-default-gateway-openshift-default-778b7b5c47-km48c\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.676276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.676233    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:17.956970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.956944    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 03 23:38:17.959515 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:38:17.959488    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda01ba_ca83_4671_8e3c_b2ffc559f992.slice/crio-0dc551c1e2b7da7a031f3983df69fc25f2a5d2378418b98e74adca868e7e4987 WatchSource:0}: Error finding container 0dc551c1e2b7da7a031f3983df69fc25f2a5d2378418b98e74adca868e7e4987: Status 404 returned error can't find the container with id 0dc551c1e2b7da7a031f3983df69fc25f2a5d2378418b98e74adca868e7e4987
Nov 03 23:38:17.961777 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.961738    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:17.961877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.961821    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:17.961877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:17.961865    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:38:18.354017 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:18.353979    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" event={"ID":"dcda01ba-ca83-4671-8e3c-b2ffc559f992","Type":"ContainerStarted","Data":"f1aca37da1818d8a06da9f5f826890d9a0dbdbe315df15695e2be80ff2ccbfc9"}
Nov 03 23:38:18.354475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:18.354032    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" event={"ID":"dcda01ba-ca83-4671-8e3c-b2ffc559f992","Type":"ContainerStarted","Data":"0dc551c1e2b7da7a031f3983df69fc25f2a5d2378418b98e74adca868e7e4987"}
Nov 03 23:38:18.383264 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:18.383162    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podStartSLOduration=1.3831389139999999 podStartE2EDuration="1.383138914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:18.380811918 +0000 UTC m=+514.271873856" watchObservedRunningTime="2026-11-03 23:38:18.383138914 +0000 UTC m=+514.274200929"
Nov 03 23:38:18.676503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:18.676415    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:19.678270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:19.678238    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:19.678709 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:19.678289    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:20.677147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:20.677111    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:20.677329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:20.677169    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:21.677227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:21.677184    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:21.677742 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:21.677247    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:22.676621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:22.676588    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:22.676807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:22.676643    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:23.676700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:23.676656    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:23.677134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:23.676719    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:24.676552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:24.676504    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 03 23:38:24.676742 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:24.676587    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 03 23:38:26.292055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:26.292010    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:26.292599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:26.292431    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:26.293624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:38:26.293597    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 03 23:38:34.830591 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:38:34.830488    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:39:04.829929 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:39:04.829848    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:39:11.988768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:11.988734    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:39:11.992424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:11.992382    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.006066 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.006037    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e9be6c8-openshift-default-dockercfg-9jcnc\""
Nov 03 23:39:12.012756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.012731    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:39:12.042096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042065    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042108    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042133    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm625\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042289    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042331    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042352    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042426    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042442    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.042528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.042456    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.143852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.143812    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.143878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.143921    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm625\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144002    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144043    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144074    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144120    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144144    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144169    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144320    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144337    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144640 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144614    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144642    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.144821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.144799    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.146442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.146419    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.146630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.146614    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.157342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.157308    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.157468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.157347    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm625\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\") pod \"e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") " pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.305574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.305541    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:12.594720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.594564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:39:12.596464 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:12.596431    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b937915_9ff0_4fd5_bc87_9ed5b84120ad.slice/crio-45eaa0fae979905ac37099b777a38bb2fd14e2754ba33772a1cdae2a905b5fdf WatchSource:0}: Error finding container 45eaa0fae979905ac37099b777a38bb2fd14e2754ba33772a1cdae2a905b5fdf: Status 404 returned error can't find the container with id 45eaa0fae979905ac37099b777a38bb2fd14e2754ba33772a1cdae2a905b5fdf
Nov 03 23:39:12.598597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.598566    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:12.598701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.598645    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:12.598701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.598690    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:12.611628 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:12.611608    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerStarted","Data":"45eaa0fae979905ac37099b777a38bb2fd14e2754ba33772a1cdae2a905b5fdf"}
Nov 03 23:39:13.617832 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:13.617796    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerStarted","Data":"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"}
Nov 03 23:39:13.657696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:13.657635    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podStartSLOduration=2.657619381 podStartE2EDuration="2.657619381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:13.656569548 +0000 UTC m=+569.547631496" watchObservedRunningTime="2026-11-03 23:39:13.657619381 +0000 UTC m=+569.548681316"
Nov 03 23:39:14.044199 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.044103    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:14.047815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.047792    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.064252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.064221    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7c2a2d28-serving-cert\""
Nov 03 23:39:14.064930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.064906    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:14.164613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.164583    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.164789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.164627    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.164789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.164739    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.164878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.164847    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfh2w\" (UniqueName: \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.266093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.266041    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfh2w\" (UniqueName: \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.266278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.266125    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.266278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.266176    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.266278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.266250    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.268017 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.267986    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.288265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.288238    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfh2w\" (UniqueName: \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.289472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.289433    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.295063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.294988    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\") pod \"maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.306557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.306526    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:14.311827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.311800    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:14.361476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.361446    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:14.622881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.622854    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:14.624124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.624098    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:14.640957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:14.640937    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:14.643806 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:14.643775    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d32a3d0_de19_4e48_8f2d_6d9a3f196523.slice/crio-09427c2de66048abf2f00c0cfd8493292b1c44e9bb8879c6f2dee7feac75065e WatchSource:0}: Error finding container 09427c2de66048abf2f00c0cfd8493292b1c44e9bb8879c6f2dee7feac75065e: Status 404 returned error can't find the container with id 09427c2de66048abf2f00c0cfd8493292b1c44e9bb8879c6f2dee7feac75065e
Nov 03 23:39:15.629234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:15.629196    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" event={"ID":"2d32a3d0-de19-4e48-8f2d-6d9a3f196523","Type":"ContainerStarted","Data":"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"}
Nov 03 23:39:15.629733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:15.629267    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:15.629733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:15.629290    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" event={"ID":"2d32a3d0-de19-4e48-8f2d-6d9a3f196523","Type":"ContainerStarted","Data":"09427c2de66048abf2f00c0cfd8493292b1c44e9bb8879c6f2dee7feac75065e"}
Nov 03 23:39:15.659322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:15.659267    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" podStartSLOduration=1.09361073 podStartE2EDuration="1.659252193s" totalImagesPullingTime="565.641463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:15.657503444 +0000 UTC m=+571.548565381" watchObservedRunningTime="2026-11-03 23:39:15.659252193 +0000 UTC m=+571.550314128"
Nov 03 23:39:17.822114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.822078    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:17.826294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.826271    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:17.832796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.832772    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-80942cbb-serving-cert\""
Nov 03 23:39:17.848186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.848145    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:17.915186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.915147    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:17.915186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.915190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vds\" (UniqueName: \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:17.915488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.915332    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:17.915488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:17.915440    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.020073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.020018    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.020276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.020093    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.020276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.020215    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.020276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.020247    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vds\" (UniqueName: \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.021945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.021908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.025105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.025075    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.025230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.025147    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.068467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.068431    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vds\" (UniqueName: \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\") pod \"maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.139635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.139564    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:18.221712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.221551    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:39:18.448461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.448436    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:18.450333 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:18.450299    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7ad18_dc81_4b2f_8160_8dec53211d4e.slice/crio-c697239267b7a99a411efc18da35cbfe400d7a2e2f26c894e52518f877dc5210 WatchSource:0}: Error finding container c697239267b7a99a411efc18da35cbfe400d7a2e2f26c894e52518f877dc5210: Status 404 returned error can't find the container with id c697239267b7a99a411efc18da35cbfe400d7a2e2f26c894e52518f877dc5210
Nov 03 23:39:18.645450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:18.645411    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" event={"ID":"aff7ad18-dc81-4b2f-8160-8dec53211d4e","Type":"ContainerStarted","Data":"c697239267b7a99a411efc18da35cbfe400d7a2e2f26c894e52518f877dc5210"}
Nov 03 23:39:19.651162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:19.651121    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" event={"ID":"aff7ad18-dc81-4b2f-8160-8dec53211d4e","Type":"ContainerStarted","Data":"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"}
Nov 03 23:39:19.651563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:19.651204    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:19.694348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:19.694296    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" podStartSLOduration=2.269800359 podStartE2EDuration="2.694282396s" totalImagesPullingTime="424.482037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:19.692038481 +0000 UTC m=+575.583100417" watchObservedRunningTime="2026-11-03 23:39:19.694282396 +0000 UTC m=+575.585344350"
Nov 03 23:39:20.770075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.770040    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:39:20.774215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.773759    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.803862 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.803832    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:39:20.833889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.833865    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d3157179-gw-openshift-default-dockercfg-mwc54\""
Nov 03 23:39:20.858566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858535    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858571    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khsl\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858608    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858683    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858786    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858811    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858915    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.858970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.858955    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.859069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.859054    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961439    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961497    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961639    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961673    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khsl\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961729    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961758    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961792    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.961822    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.962686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.962258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.972158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.972124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.976761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.972670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.976761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.973230    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.976761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.973759    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.984269 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.982127    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.989020 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.988993    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.998655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.998618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:20.998903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:20.998876    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khsl\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\") pod \"e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") " pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:21.087371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.087337    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:21.379220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.379188    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:39:21.380986 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:21.380961    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f94534_eaaf_4282_86c7_1a9844db61fe.slice/crio-af342f1d7c302401f57cf22271f216b1c2c26a8d51d01392b9517a94a1bbafc4 WatchSource:0}: Error finding container af342f1d7c302401f57cf22271f216b1c2c26a8d51d01392b9517a94a1bbafc4: Status 404 returned error can't find the container with id af342f1d7c302401f57cf22271f216b1c2c26a8d51d01392b9517a94a1bbafc4
Nov 03 23:39:21.383263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.383231    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:21.383346 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.383292    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:21.383346 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.383325    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:21.638939 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.638865    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:21.663523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.663478    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerStarted","Data":"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"}
Nov 03 23:39:21.663692 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.663549    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerStarted","Data":"af342f1d7c302401f57cf22271f216b1c2c26a8d51d01392b9517a94a1bbafc4"}
Nov 03 23:39:21.720285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:21.720217    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podStartSLOduration=1.7202009619999998 podStartE2EDuration="1.720200962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:21.719256191 +0000 UTC m=+577.610318154" watchObservedRunningTime="2026-11-03 23:39:21.720200962 +0000 UTC m=+577.611262897"
Nov 03 23:39:22.087620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:22.087575    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:22.093147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:22.093122    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:22.668794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:22.668763    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:22.669835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:22.669816    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:25.661339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:25.661303    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:25.923647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:25.923570    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:39:29.228898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.228861    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:39:29.242376 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.242344    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.251147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.250791    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-044f4dcd-serving-cert\""
Nov 03 23:39:29.262797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.262768    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:39:29.331989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.331947    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.332523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.332185    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.332523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.332230    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghvb\" (UniqueName: \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.332523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.332308    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.433868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.433814    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.434149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.434122    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghvb\" (UniqueName: \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.434344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.434323    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.434604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.434582    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.435261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.435235    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.437846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.437826    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.438108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.438088    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.449007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.448983    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghvb\" (UniqueName: \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\") pod \"maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.557751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.557712    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:29.879641 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:29.879574    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:39:29.882538 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:29.882502    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f16b4c9_21ea_42a1_b09c_4d19128a0178.slice/crio-2aef9fa54c56dcf2ce6fbb775600990dee428806a0f9f2673e353e338b082755 WatchSource:0}: Error finding container 2aef9fa54c56dcf2ce6fbb775600990dee428806a0f9f2673e353e338b082755: Status 404 returned error can't find the container with id 2aef9fa54c56dcf2ce6fbb775600990dee428806a0f9f2673e353e338b082755
Nov 03 23:39:30.739292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:30.739178    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerStarted","Data":"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"}
Nov 03 23:39:30.739807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:30.739425    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:30.739807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:30.739466    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerStarted","Data":"2aef9fa54c56dcf2ce6fbb775600990dee428806a0f9f2673e353e338b082755"}
Nov 03 23:39:30.768370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:30.768320    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" podStartSLOduration=1.28587313 podStartE2EDuration="1.768307535s" totalImagesPullingTime="482.434405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:30.767636826 +0000 UTC m=+586.658698762" watchObservedRunningTime="2026-11-03 23:39:30.768307535 +0000 UTC m=+586.659369514"
Nov 03 23:39:32.791408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:32.791314    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:39:34.852964 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:39:34.852924    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:39:36.745775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.745731    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.37:8443/health\": dial tcp 10.134.0.37:8443: connect: connection refused"
Nov 03 23:39:36.773950 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.773919    6430 generic.go:356] "Generic (PLEG): container finished" podID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerID="4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9" exitCode=0
Nov 03 23:39:36.774103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.773972    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerDied","Data":"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"}
Nov 03 23:39:36.774382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.774358    6430 scope.go:122] "RemoveContainer" containerID="4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"
Nov 03 23:39:36.775702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.775672    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:36.775778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.775741    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:36.775778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:36.775773    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:37.781052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:37.781022    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerStarted","Data":"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"}
Nov 03 23:39:38.306208 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:38.306166    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:38.306497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:38.306465    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 23:39:38.306716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:38.306684    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 23:39:39.078200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.078166    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:39:39.081852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.081815    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.095087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.095058    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d3157179-serving-cert\""
Nov 03 23:39:39.097307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.097282    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:39:39.150360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.150322    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4t7\" (UniqueName: \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.150360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.150358    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.150639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.150444    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.150639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.150495    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.251543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.251511    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4t7\" (UniqueName: \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.251543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.251548    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.251798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.251573    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.251798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.251609    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.252294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.252265    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.254046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.254017    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.254159 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.254147    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.264977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.264952    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4t7\" (UniqueName: \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\") pod \"maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.311412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.308194    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 23:39:39.318537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.318495    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 23:39:39.396579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.396491    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:39.731878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.730119    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:39:39.734373 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:39.734340    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e28aef7_f67c_4b22_ba12_e291016d5d91.slice/crio-e63761bb006de599a29df9a6ab8d50d5b7b4f1943a0a02941203fb5a9aba0148 WatchSource:0}: Error finding container e63761bb006de599a29df9a6ab8d50d5b7b4f1943a0a02941203fb5a9aba0148: Status 404 returned error can't find the container with id e63761bb006de599a29df9a6ab8d50d5b7b4f1943a0a02941203fb5a9aba0148
Nov 03 23:39:39.793238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:39.793202    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" event={"ID":"8e28aef7-f67c-4b22-ba12-e291016d5d91","Type":"ContainerStarted","Data":"e63761bb006de599a29df9a6ab8d50d5b7b4f1943a0a02941203fb5a9aba0148"}
Nov 03 23:39:40.306510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.306477    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 23:39:40.307006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.306975    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 23:39:40.799508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.799478    6430 generic.go:356] "Generic (PLEG): container finished" podID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerID="82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5" exitCode=1
Nov 03 23:39:40.799658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.799552    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerDied","Data":"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"}
Nov 03 23:39:40.799982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.799966    6430 scope.go:122] "RemoveContainer" containerID="82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"
Nov 03 23:39:40.801454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.801189    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" event={"ID":"8e28aef7-f67c-4b22-ba12-e291016d5d91","Type":"ContainerStarted","Data":"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"}
Nov 03 23:39:40.801454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.801242    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:40.849842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:40.849787    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" podStartSLOduration=1.386375704 podStartE2EDuration="1.849771916s" totalImagesPullingTime="463.396212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:40.848756957 +0000 UTC m=+596.739818892" watchObservedRunningTime="2026-11-03 23:39:40.849771916 +0000 UTC m=+596.740833854"
Nov 03 23:39:41.329415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:41.306461    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 23:39:41.329415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:41.306521    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 23:39:41.633798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:41.633758    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:39:41.820417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:41.817178    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerStarted","Data":"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"}
Nov 03 23:39:41.820417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:41.818030    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:42.313413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:42.306828    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 23:39:42.313413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:42.306892    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 23:39:43.331816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:43.331779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:43.333465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:43.333406    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:43.338247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:43.338180    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:39:46.837272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:46.837240    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:39:47.845658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:47.845628    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:39:48.292602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.292526    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:48.292789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.292766    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" podUID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" containerName="maas-api" containerID="cri-o://8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed" gracePeriod=30
Nov 03 23:39:48.583702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.583656    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:48.691514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.691478    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\") pod \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") "
Nov 03 23:39:48.691660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.691595    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\") pod \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") "
Nov 03 23:39:48.691660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.691630    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\") pod \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") "
Nov 03 23:39:48.691660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.691656    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\" (UniqueName: \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\") pod \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\" (UID: \"aff7ad18-dc81-4b2f-8160-8dec53211d4e\") "
Nov 03 23:39:48.691908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.691880    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca" pod "aff7ad18-dc81-4b2f-8160-8dec53211d4e" (UID: "aff7ad18-dc81-4b2f-8160-8dec53211d4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:48.692098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.692071    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff7ad18-dc81-4b2f-8160-8dec53211d4e-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:48.693970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.693934    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds" pod "aff7ad18-dc81-4b2f-8160-8dec53211d4e" (UID: "aff7ad18-dc81-4b2f-8160-8dec53211d4e"). InnerVolumeSpecName "kube-api-access-s6vds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:48.694078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.694030    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls" pod "aff7ad18-dc81-4b2f-8160-8dec53211d4e" (UID: "aff7ad18-dc81-4b2f-8160-8dec53211d4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:48.694137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.694109    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs" pod "aff7ad18-dc81-4b2f-8160-8dec53211d4e" (UID: "aff7ad18-dc81-4b2f-8160-8dec53211d4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:48.792883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.792845    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vds\" (UniqueName: \"kubernetes.io/projected/aff7ad18-dc81-4b2f-8160-8dec53211d4e-kube-api-access-s6vds\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:48.792883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.792877    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:48.792883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.792891    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff7ad18-dc81-4b2f-8160-8dec53211d4e-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:48.871153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.871070    6430 generic.go:356] "Generic (PLEG): container finished" podID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" containerID="8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed" exitCode=0
Nov 03 23:39:48.871153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.871123    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"
Nov 03 23:39:48.871153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.871129    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" event={"ID":"aff7ad18-dc81-4b2f-8160-8dec53211d4e","Type":"ContainerDied","Data":"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"}
Nov 03 23:39:48.871710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.871177    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l" event={"ID":"aff7ad18-dc81-4b2f-8160-8dec53211d4e","Type":"ContainerDied","Data":"c697239267b7a99a411efc18da35cbfe400d7a2e2f26c894e52518f877dc5210"}
Nov 03 23:39:48.871710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.871200    6430 scope.go:122] "RemoveContainer" containerID="8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"
Nov 03 23:39:48.881965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.881945    6430 scope.go:122] "RemoveContainer" containerID="8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"
Nov 03 23:39:48.882287 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:39:48.882267    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed\": container with ID starting with 8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed not found: ID does not exist" containerID="8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"
Nov 03 23:39:48.882344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.882299    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed"} err="failed to get container status \"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed\": rpc error: code = NotFound desc = could not find container \"8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed\": container with ID starting with 8d04389b086736212f1e86747c98aecd0f45cf5b4b4b104acd93c37d6e50ffed not found: ID does not exist"
Nov 03 23:39:48.916850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.916815    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:48.921579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:48.921546    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80942cbb-6555dd4b4b-8j89l"]
Nov 03 23:39:50.231316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:50.231286    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" path="/var/lib/kubelet/pods/aff7ad18-dc81-4b2f-8160-8dec53211d4e/volumes"
Nov 03 23:39:50.817115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:50.817080    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:39:50.914959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:50.914920    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:50.915218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:50.915188    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" podUID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" containerName="maas-api" containerID="cri-o://dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb" gracePeriod=30
Nov 03 23:39:51.195647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.194324    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:51.319734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.319700    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\") pod \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") "
Nov 03 23:39:51.320118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.319815    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\" (UniqueName: \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\") pod \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") "
Nov 03 23:39:51.320118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.319839    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\") pod \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") "
Nov 03 23:39:51.320118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.319872    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\") pod \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\" (UID: \"2d32a3d0-de19-4e48-8f2d-6d9a3f196523\") "
Nov 03 23:39:51.320299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.320219    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca" pod "2d32a3d0-de19-4e48-8f2d-6d9a3f196523" (UID: "2d32a3d0-de19-4e48-8f2d-6d9a3f196523"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:39:51.320717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.320693    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:51.322080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.322053    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs" pod "2d32a3d0-de19-4e48-8f2d-6d9a3f196523" (UID: "2d32a3d0-de19-4e48-8f2d-6d9a3f196523"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:51.322178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.322094    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w" pod "2d32a3d0-de19-4e48-8f2d-6d9a3f196523" (UID: "2d32a3d0-de19-4e48-8f2d-6d9a3f196523"). InnerVolumeSpecName "kube-api-access-rfh2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:39:51.322237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.322202    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls" pod "2d32a3d0-de19-4e48-8f2d-6d9a3f196523" (UID: "2d32a3d0-de19-4e48-8f2d-6d9a3f196523"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:39:51.422144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.422063    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:51.422144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.422096    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfh2w\" (UniqueName: \"kubernetes.io/projected/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-kube-api-access-rfh2w\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:51.422144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.422107    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d32a3d0-de19-4e48-8f2d-6d9a3f196523-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:39:51.890428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.890373    6430 generic.go:356] "Generic (PLEG): container finished" podID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" containerID="dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb" exitCode=0
Nov 03 23:39:51.890629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.890474    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" event={"ID":"2d32a3d0-de19-4e48-8f2d-6d9a3f196523","Type":"ContainerDied","Data":"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"}
Nov 03 23:39:51.890629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.890482    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"
Nov 03 23:39:51.890629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.890546    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7" event={"ID":"2d32a3d0-de19-4e48-8f2d-6d9a3f196523","Type":"ContainerDied","Data":"09427c2de66048abf2f00c0cfd8493292b1c44e9bb8879c6f2dee7feac75065e"}
Nov 03 23:39:51.890629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.890569    6430 scope.go:122] "RemoveContainer" containerID="dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"
Nov 03 23:39:51.892919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.892889    6430 generic.go:356] "Generic (PLEG): container finished" podID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerID="bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239" exitCode=0
Nov 03 23:39:51.893039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.892930    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerDied","Data":"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"}
Nov 03 23:39:51.893718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.893693    6430 scope.go:122] "RemoveContainer" containerID="bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"
Nov 03 23:39:51.895344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.895308    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:51.895515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.895459    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:51.895592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.895519    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:51.908461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.908439    6430 scope.go:122] "RemoveContainer" containerID="dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"
Nov 03 23:39:51.908897 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:39:51.908760    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb\": container with ID starting with dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb not found: ID does not exist" containerID="dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"
Nov 03 23:39:51.908897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.908794    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb"} err="failed to get container status \"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb\": rpc error: code = NotFound desc = could not find container \"dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb\": container with ID starting with dde09ab28722b2bdd56348851a91aa81875adde132f708d7ed83f008630ba2fb not found: ID does not exist"
Nov 03 23:39:51.999409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:51.998356    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:52.002768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.002730    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c2a2d28-6cddb97d79-sgfn7"]
Nov 03 23:39:52.026672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.026639    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 03 23:39:52.027241 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027211    6430 cpu_manager.go:414] "containerMap: removing container" podUID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" containerName="maas-api"
Nov 03 23:39:52.027417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027380    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" containerName="maas-api"
Nov 03 23:39:52.027512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027472    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" containerName="maas-api"
Nov 03 23:39:52.027512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027482    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" containerName="maas-api"
Nov 03 23:39:52.027622 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027608    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" containerName="maas-api"
Nov 03 23:39:52.027674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.027631    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff7ad18-dc81-4b2f-8160-8dec53211d4e" containerName="maas-api"
Nov 03 23:39:52.031071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.031051    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.039573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.039550    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-80f8ba-openshift-default-dockercfg-rx8cf\""
Nov 03 23:39:52.056294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.056262    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 03 23:39:52.131076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131082    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131172    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131351    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131428    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131468    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mww7\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131592    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131657    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.131725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.131710    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.232935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.232837    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.232935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.232909    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.232959    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.232996    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233055    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233088    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233215    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233254    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mww7\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233675    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.233961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.233945    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.234435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.234377    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.234621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.234532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.234844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.234817    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.241833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.241803    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d32a3d0-de19-4e48-8f2d-6d9a3f196523" path="/var/lib/kubelet/pods/2d32a3d0-de19-4e48-8f2d-6d9a3f196523/volumes"
Nov 03 23:39:52.266808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.266736    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.267085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.267054    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.277411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.273683    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mww7\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.278122 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.278072    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.346430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.346375    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:52.669593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.669558    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:52.675129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.675103    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 03 23:39:52.676605 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:39:52.676578    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d46e516_002a_49cb_87b0_c04311b0f809.slice/crio-6dbc5d315d01cc9ea135e61a30980cb3d525e0ffc53051d96fea32dc50e0de46 WatchSource:0}: Error finding container 6dbc5d315d01cc9ea135e61a30980cb3d525e0ffc53051d96fea32dc50e0de46: Status 404 returned error can't find the container with id 6dbc5d315d01cc9ea135e61a30980cb3d525e0ffc53051d96fea32dc50e0de46
Nov 03 23:39:52.678966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.678935    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:52.679046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.678998    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:52.679046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.679031    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:39:52.906138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.906095    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerStarted","Data":"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"}
Nov 03 23:39:52.931767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.930631    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerStarted","Data":"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"}
Nov 03 23:39:52.931767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.930706    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerStarted","Data":"6dbc5d315d01cc9ea135e61a30980cb3d525e0ffc53051d96fea32dc50e0de46"}
Nov 03 23:39:52.971196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:52.971142    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podStartSLOduration=1.971127213 podStartE2EDuration="1.971127213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:52.970286838 +0000 UTC m=+608.861348775" watchObservedRunningTime="2026-11-03 23:39:52.971127213 +0000 UTC m=+608.862189208"
Nov 03 23:39:53.090230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.090134    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:53.090450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.090411    6430 patch_prober.go:29] interesting pod/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 23:39:53.090531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.090483    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 23:39:53.347138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.347098    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:53.352924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.352897    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:53.936329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.936084    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:53.936846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:53.936830    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:39:54.088526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:54.088495    6430 patch_prober.go:29] interesting pod/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 23:39:54.088696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:54.088548    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 23:39:55.087980 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:55.087938    6430 patch_prober.go:29] interesting pod/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 23:39:55.088411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:55.088011    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 23:39:56.087840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:56.087806    6430 patch_prober.go:29] interesting pod/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 23:39:56.088042 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:56.087870    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 23:39:57.088449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:57.088384    6430 patch_prober.go:29] interesting pod/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 23:39:57.088943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:57.088481    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 23:39:57.404045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:57.403959    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:39:57.404222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:57.404200    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" containerID="cri-o://c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88" gracePeriod=30
Nov 03 23:39:58.091988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:58.091955    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:58.092411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:58.092255    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:58.092977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:58.092960    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:39:58.335900 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:58.335863    6430 patch_prober.go:29] interesting pod/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:39:58.336076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:58.335935    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:39:59.513884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:39:59.513845    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:02.596795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.596767    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:40:02.679640 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679542    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679640 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679640 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679628    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679754    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679806    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679857    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.679924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679884    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.680129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.679954    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.680129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680096    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\") pod \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\" (UID: \"6b937915-9ff0-4fd5-bc87-9ed5b84120ad\") "
Nov 03 23:40:02.680482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680410    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:02.680647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680518    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.680647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:02.680807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680738    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:02.680904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.680870    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:02.681229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.681205    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:02.682008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.681980    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:40:02.682567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.682542    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "kube-api-access-nm625". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:02.682864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.682833    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:02.683139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.683118    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy" pod "6b937915-9ff0-4fd5-bc87-9ed5b84120ad" (UID: "6b937915-9ff0-4fd5-bc87-9ed5b84120ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:02.780974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.780943    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.780974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.780974    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm625\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-kube-api-access-nm625\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.780974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.780987    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.781240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.781000    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.781240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.781012    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.781240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.781023    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.781240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.781035    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.781240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.781046    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b937915-9ff0-4fd5-bc87-9ed5b84120ad-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:02.986135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.986043    6430 generic.go:356] "Generic (PLEG): container finished" podID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerID="c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88" exitCode=0
Nov 03 23:40:02.986135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.986116    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"
Nov 03 23:40:02.986135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.986119    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerDied","Data":"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"}
Nov 03 23:40:02.986364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.986201    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8" event={"ID":"6b937915-9ff0-4fd5-bc87-9ed5b84120ad","Type":"ContainerDied","Data":"45eaa0fae979905ac37099b777a38bb2fd14e2754ba33772a1cdae2a905b5fdf"}
Nov 03 23:40:02.986364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.986224    6430 scope.go:122] "RemoveContainer" containerID="c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"
Nov 03 23:40:02.995923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:02.995904    6430 scope.go:122] "RemoveContainer" containerID="4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"
Nov 03 23:40:03.005347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.005323    6430 scope.go:122] "RemoveContainer" containerID="c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"
Nov 03 23:40:03.005831 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:03.005804    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88\": container with ID starting with c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88 not found: ID does not exist" containerID="c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"
Nov 03 23:40:03.005948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.005840    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88"} err="failed to get container status \"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88\": rpc error: code = NotFound desc = could not find container \"c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88\": container with ID starting with c59e32002f2e251f3162b237ec56348d65961c57a277a164b1b4c84d54716f88 not found: ID does not exist"
Nov 03 23:40:03.005948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.005867    6430 scope.go:122] "RemoveContainer" containerID="4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"
Nov 03 23:40:03.006169 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:03.006145    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9\": container with ID starting with 4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9 not found: ID does not exist" containerID="4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"
Nov 03 23:40:03.006255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.006176    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9"} err="failed to get container status \"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9\": rpc error: code = NotFound desc = could not find container \"4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9\": container with ID starting with 4771f12931c56537c2e235e805aef32c72acbc0f6d363ae4d4a5672f9878d1d9 not found: ID does not exist"
Nov 03 23:40:03.060151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.060111    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:40:03.060362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.060340    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api" containerID="cri-o://9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7" gracePeriod=30
Nov 03 23:40:03.061547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.061525    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:40:03.103594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.103562    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e9be6c8-openshift-default-7784768447-lf8m8"]
Nov 03 23:40:03.316431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.313712    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:40:03.390097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.390062    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\") pod \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") "
Nov 03 23:40:03.390274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.390117    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\") pod \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") "
Nov 03 23:40:03.390274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.390176    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\" (UniqueName: \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\") pod \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") "
Nov 03 23:40:03.390274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.390237    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\") pod \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\" (UID: \"0f16b4c9-21ea-42a1-b09c-4d19128a0178\") "
Nov 03 23:40:03.390546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.390514    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca" pod "0f16b4c9-21ea-42a1-b09c-4d19128a0178" (UID: "0f16b4c9-21ea-42a1-b09c-4d19128a0178"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:03.392307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.392283    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs" pod "0f16b4c9-21ea-42a1-b09c-4d19128a0178" (UID: "0f16b4c9-21ea-42a1-b09c-4d19128a0178"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:40:03.392402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.392305    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls" pod "0f16b4c9-21ea-42a1-b09c-4d19128a0178" (UID: "0f16b4c9-21ea-42a1-b09c-4d19128a0178"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:40:03.392402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.392341    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb" pod "0f16b4c9-21ea-42a1-b09c-4d19128a0178" (UID: "0f16b4c9-21ea-42a1-b09c-4d19128a0178"). InnerVolumeSpecName "kube-api-access-vghvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:03.491639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.491609    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:03.491639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.491638    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f16b4c9-21ea-42a1-b09c-4d19128a0178-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:03.491639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.491648    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f16b4c9-21ea-42a1-b09c-4d19128a0178-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:03.491881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.491657    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghvb\" (UniqueName: \"kubernetes.io/projected/0f16b4c9-21ea-42a1-b09c-4d19128a0178-kube-api-access-vghvb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:03.681489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.681379    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 03 23:40:03.682049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682027    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682052    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682069    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682076    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682092    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682100    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682118    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.682136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682127    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.682479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682242    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.682479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682263    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:03.682479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.682273    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" containerName="istio-proxy"
Nov 03 23:40:03.685741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.685718    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693090    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693150    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kccc4\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693306 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693258    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693311    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693427    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693463    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693481    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693591    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.693664 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.693627    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.722676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.722636    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\""
Nov 03 23:40:03.725112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.725081    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 03 23:40:03.794158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794114    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794165    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794208    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kccc4\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794245    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794272    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794318    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794348    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794362    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794429    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794740    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794774    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794802    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.794915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.794861    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.795152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.795132    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.796614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.796591    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.796959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.796938    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.825322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.825290    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:03.826548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:03.826529    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kccc4\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:04.000894 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.000812    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:04.002075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.002047    6430 generic.go:356] "Generic (PLEG): container finished" podID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerID="9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7" exitCode=0
Nov 03 23:40:04.002223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.002103    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"
Nov 03 23:40:04.002223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.002140    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerDied","Data":"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"}
Nov 03 23:40:04.002328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.002226    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8" event={"ID":"0f16b4c9-21ea-42a1-b09c-4d19128a0178","Type":"ContainerDied","Data":"2aef9fa54c56dcf2ce6fbb775600990dee428806a0f9f2673e353e338b082755"}
Nov 03 23:40:04.002328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.002275    6430 scope.go:122] "RemoveContainer" containerID="9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"
Nov 03 23:40:04.013484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.013461    6430 scope.go:122] "RemoveContainer" containerID="82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"
Nov 03 23:40:04.035193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.035169    6430 scope.go:122] "RemoveContainer" containerID="9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"
Nov 03 23:40:04.035581 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:04.035562    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7\": container with ID starting with 9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7 not found: ID does not exist" containerID="9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"
Nov 03 23:40:04.035631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.035592    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7"} err="failed to get container status \"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7\": rpc error: code = NotFound desc = could not find container \"9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7\": container with ID starting with 9476d96c0a9e536923b96d3543195547d3a0c787519bd6900c17378d8531b2d7 not found: ID does not exist"
Nov 03 23:40:04.035631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.035613    6430 scope.go:122] "RemoveContainer" containerID="82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"
Nov 03 23:40:04.035928 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:04.035911    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5\": container with ID starting with 82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5 not found: ID does not exist" containerID="82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"
Nov 03 23:40:04.035981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.035937    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5"} err="failed to get container status \"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5\": rpc error: code = NotFound desc = could not find container \"82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5\": container with ID starting with 82b55704af67de98e246528ee031eed55c13c3bbc2cc2f2282358bed1d5b75f5 not found: ID does not exist"
Nov 03 23:40:04.049364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.049338    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:40:04.054666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.054641    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-044f4dcd-794dfc97c4-f8pg8"]
Nov 03 23:40:04.230007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.229972    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" path="/var/lib/kubelet/pods/0f16b4c9-21ea-42a1-b09c-4d19128a0178/volumes"
Nov 03 23:40:04.230542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.230521    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b937915-9ff0-4fd5-bc87-9ed5b84120ad" path="/var/lib/kubelet/pods/6b937915-9ff0-4fd5-bc87-9ed5b84120ad/volumes"
Nov 03 23:40:04.372476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.372443    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 03 23:40:04.374649 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:04.374622    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e74a34_036b_4930_b8df_4219c4cfb09c.slice/crio-799eea3db3b8c735c27a4311eafa43f5254d31b48a3bb4fb26fb37dba7cd8b3e WatchSource:0}: Error finding container 799eea3db3b8c735c27a4311eafa43f5254d31b48a3bb4fb26fb37dba7cd8b3e: Status 404 returned error can't find the container with id 799eea3db3b8c735c27a4311eafa43f5254d31b48a3bb4fb26fb37dba7cd8b3e
Nov 03 23:40:04.377243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.377212    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:04.377318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.377290    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:04.377351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:04.377326    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:04.873340 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:04.873305    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:40:05.034651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:05.034608    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerStarted","Data":"b828cb530ef173a45db5398a5088d3dc061579fe2084445e6b7a650e46d885fc"}
Nov 03 23:40:05.034823 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:05.034671    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerStarted","Data":"799eea3db3b8c735c27a4311eafa43f5254d31b48a3bb4fb26fb37dba7cd8b3e"}
Nov 03 23:40:06.002077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:06.002041    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:06.027830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:06.027781    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:06.064438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:06.064162    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:06.065948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:06.065923    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:06.134058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:06.133995    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podStartSLOduration=3.133973821 podStartE2EDuration="3.133973821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:05.129707958 +0000 UTC m=+621.020769893" watchObservedRunningTime="2026-11-03 23:40:06.133973821 +0000 UTC m=+622.025035761"
Nov 03 23:40:08.682215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:08.682180    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:12.127292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.127254    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:40:12.127830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.127496    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" podUID="8e28aef7-f67c-4b22-ba12-e291016d5d91" containerName="maas-api" containerID="cri-o://0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454" gracePeriod=30
Nov 03 23:40:12.414218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.414190    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:40:12.577804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.577766    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\") pod \"8e28aef7-f67c-4b22-ba12-e291016d5d91\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") "
Nov 03 23:40:12.578005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.577923    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\" (UniqueName: \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\") pod \"8e28aef7-f67c-4b22-ba12-e291016d5d91\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") "
Nov 03 23:40:12.578005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.577957    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\") pod \"8e28aef7-f67c-4b22-ba12-e291016d5d91\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") "
Nov 03 23:40:12.578125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.578056    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\") pod \"8e28aef7-f67c-4b22-ba12-e291016d5d91\" (UID: \"8e28aef7-f67c-4b22-ba12-e291016d5d91\") "
Nov 03 23:40:12.578520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.578485    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca" pod "8e28aef7-f67c-4b22-ba12-e291016d5d91" (UID: "8e28aef7-f67c-4b22-ba12-e291016d5d91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:12.580529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.580492    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs" pod "8e28aef7-f67c-4b22-ba12-e291016d5d91" (UID: "8e28aef7-f67c-4b22-ba12-e291016d5d91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:40:12.580879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.580847    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls" pod "8e28aef7-f67c-4b22-ba12-e291016d5d91" (UID: "8e28aef7-f67c-4b22-ba12-e291016d5d91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:40:12.581077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.581050    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7" pod "8e28aef7-f67c-4b22-ba12-e291016d5d91" (UID: "8e28aef7-f67c-4b22-ba12-e291016d5d91"). InnerVolumeSpecName "kube-api-access-kg4t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:12.679961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.679878    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e28aef7-f67c-4b22-ba12-e291016d5d91-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:12.679961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.679911    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:12.679961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.679925    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg4t7\" (UniqueName: \"kubernetes.io/projected/8e28aef7-f67c-4b22-ba12-e291016d5d91-kube-api-access-kg4t7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:12.679961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:12.679934    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e28aef7-f67c-4b22-ba12-e291016d5d91-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:13.099833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.099793    6430 generic.go:356] "Generic (PLEG): container finished" podID="8e28aef7-f67c-4b22-ba12-e291016d5d91" containerID="0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454" exitCode=0
Nov 03 23:40:13.100036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.099852    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"
Nov 03 23:40:13.100036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.099886    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" event={"ID":"8e28aef7-f67c-4b22-ba12-e291016d5d91","Type":"ContainerDied","Data":"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"}
Nov 03 23:40:13.100036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.099957    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp" event={"ID":"8e28aef7-f67c-4b22-ba12-e291016d5d91","Type":"ContainerDied","Data":"e63761bb006de599a29df9a6ab8d50d5b7b4f1943a0a02941203fb5a9aba0148"}
Nov 03 23:40:13.100036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.099989    6430 scope.go:122] "RemoveContainer" containerID="0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"
Nov 03 23:40:13.109853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.109830    6430 scope.go:122] "RemoveContainer" containerID="0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"
Nov 03 23:40:13.110129 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:13.110111    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454\": container with ID starting with 0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454 not found: ID does not exist" containerID="0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"
Nov 03 23:40:13.110188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.110140    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454"} err="failed to get container status \"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454\": rpc error: code = NotFound desc = could not find container \"0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454\": container with ID starting with 0d52cf49f75b28434e260bf990011b1f21700eeb851d156c916242b1cd218454 not found: ID does not exist"
Nov 03 23:40:13.263424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.263375    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:40:13.269007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.268975    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3157179-f46ddcff4-pqcnp"]
Nov 03 23:40:13.415981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.415905    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:40:13.416152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:13.416127    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy" containerID="cri-o://4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6" gracePeriod=30
Nov 03 23:40:14.107053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.107019    6430 generic.go:356] "Generic (PLEG): container finished" podID="0d46e516-002a-49cb-87b0-c04311b0f809" containerID="a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c" exitCode=0
Nov 03 23:40:14.107270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.107092    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerDied","Data":"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"}
Nov 03 23:40:14.107536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.107522    6430 scope.go:122] "RemoveContainer" containerID="a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"
Nov 03 23:40:14.108782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.108757    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:14.108872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.108825    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:14.108872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.108859    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:14.231097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.231064    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e28aef7-f67c-4b22-ba12-e291016d5d91" path="/var/lib/kubelet/pods/8e28aef7-f67c-4b22-ba12-e291016d5d91/volumes"
Nov 03 23:40:14.561856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.561823    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 03 23:40:14.562461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.562441    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8e28aef7-f67c-4b22-ba12-e291016d5d91" containerName="maas-api"
Nov 03 23:40:14.562559 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.562463    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28aef7-f67c-4b22-ba12-e291016d5d91" containerName="maas-api"
Nov 03 23:40:14.562630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.562616    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e28aef7-f67c-4b22-ba12-e291016d5d91" containerName="maas-api"
Nov 03 23:40:14.562680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.562641    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f16b4c9-21ea-42a1-b09c-4d19128a0178" containerName="maas-api"
Nov 03 23:40:14.564712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.564689    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.634428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.632556    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-5738a9-serving-cert\""
Nov 03 23:40:14.637463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.635229    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 03 23:40:14.702959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.702917    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.703134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.702959    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zl5\" (UniqueName: \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.703134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.703066    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.703134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.703104    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.804337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.804293    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.804337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.804352    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.804639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.804450    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.804639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.804472    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zl5\" (UniqueName: \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.805095 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.805071    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.821492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.819113    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.834417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.822552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.854850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.854813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zl5\" (UniqueName: \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\") pod \"maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:14.881452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:14.881293    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:15.152553 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.152304    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerStarted","Data":"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"}
Nov 03 23:40:15.229055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.229014    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 03 23:40:15.239679 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:15.239650    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbb7e6f_eec4_41cf_9ad4_e7879d0d512f.slice/crio-b1765df54b9b93012d54679690f620bf2b2e3a83fc267ab1b9d9baaad1509f94 WatchSource:0}: Error finding container b1765df54b9b93012d54679690f620bf2b2e3a83fc267ab1b9d9baaad1509f94: Status 404 returned error can't find the container with id b1765df54b9b93012d54679690f620bf2b2e3a83fc267ab1b9d9baaad1509f94
Nov 03 23:40:15.244897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.244221    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:40:15.347073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.347029    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:40:15.347310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.347285    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 23:40:15.347376 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:15.347354    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 23:40:16.159040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.158998    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" event={"ID":"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f","Type":"ContainerStarted","Data":"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"}
Nov 03 23:40:16.159588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.159149    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:16.159588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.159186    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" event={"ID":"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f","Type":"ContainerStarted","Data":"b1765df54b9b93012d54679690f620bf2b2e3a83fc267ab1b9d9baaad1509f94"}
Nov 03 23:40:16.237844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.237786    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" podStartSLOduration=1.775367567 podStartE2EDuration="2.237771523s" totalImagesPullingTime="462.403956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:16.236633208 +0000 UTC m=+632.127695142" watchObservedRunningTime="2026-11-03 23:40:16.237771523 +0000 UTC m=+632.128833458"
Nov 03 23:40:16.347475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.347431    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 23:40:16.347646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.347506    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 23:40:16.733271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.733215    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 03 23:40:16.737849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.737825    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.757665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.757644    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5cb349-openshift-default-dockercfg-qzhc9\""
Nov 03 23:40:16.757998 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.757971    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758023    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758162    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758275    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758365    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758427    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758623    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj6q\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758690    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.758857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.758791    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.763110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.763088    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 03 23:40:16.859572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859520    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.859772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859600    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.859772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859641    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.859772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859679    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj6q\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.859772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859711    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.859772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859738    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859798    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859838    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.859888    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.860091    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.860260    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.860686    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.860966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.860945    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.861074 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.860975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.863448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.863423    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.863693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.863672    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.883870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.883824    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj6q\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:16.884383 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:16.884358    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:17.050915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.050624    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:17.347035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.346954    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 23:40:17.347035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.347003    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 23:40:17.431411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.431356    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 03 23:40:17.434510 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:17.434478    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d029757_57ec_4103_8a27_f7ab436a2c3b.slice/crio-4981e0bb0cdcec40eb467c310ae423a64abeb778d9ee702f0c2b270aa165021c WatchSource:0}: Error finding container 4981e0bb0cdcec40eb467c310ae423a64abeb778d9ee702f0c2b270aa165021c: Status 404 returned error can't find the container with id 4981e0bb0cdcec40eb467c310ae423a64abeb778d9ee702f0c2b270aa165021c
Nov 03 23:40:17.437658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.437629    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:17.437734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.437698    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:17.437734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.437726    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:17.755833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:17.755742    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:18.172266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.172232    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerStarted","Data":"266c03d8800ce3d9c0f88022f87141afa4e18ec23044ab9cbec79bf6f5a531db"}
Nov 03 23:40:18.172476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.172289    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerStarted","Data":"4981e0bb0cdcec40eb467c310ae423a64abeb778d9ee702f0c2b270aa165021c"}
Nov 03 23:40:18.237174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.237121    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podStartSLOduration=2.237106802 podStartE2EDuration="2.237106802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:18.234819779 +0000 UTC m=+634.125881715" watchObservedRunningTime="2026-11-03 23:40:18.237106802 +0000 UTC m=+634.128168736"
Nov 03 23:40:18.347582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.347534    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 23:40:18.348072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.347605    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 23:40:18.890220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.890189    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:40:18.994718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.994907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994764    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.994907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994807    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.994907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994857    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.994907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994883    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.995064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994902    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.995064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994944    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.995064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994961    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.995064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.994990    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\") pod \"23f94534-eaaf-4282-86c7-1a9844db61fe\" (UID: \"23f94534-eaaf-4282-86c7-1a9844db61fe\") "
Nov 03 23:40:18.995064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995004    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:18.995347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995328    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:18.995430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995363    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:18.995634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995615    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:18.995869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995838    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:18.995959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.995926    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:40:18.997711 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.997687    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "kube-api-access-6khsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:18.997806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.997727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:40:18.997953 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.997921    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:40:18.998040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:18.998021    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo" pod "23f94534-eaaf-4282-86c7-1a9844db61fe" (UID: "23f94534-eaaf-4282-86c7-1a9844db61fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:40:19.060890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.060662    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:19.065847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.065818    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:19.096506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096472    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f94534-eaaf-4282-86c7-1a9844db61fe-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096497    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096507    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096515    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096526    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096535    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6khsl\" (UniqueName: \"kubernetes.io/projected/23f94534-eaaf-4282-86c7-1a9844db61fe-kube-api-access-6khsl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096544    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.096774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.096553    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f94534-eaaf-4282-86c7-1a9844db61fe-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:40:19.178616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.178579    6430 generic.go:356] "Generic (PLEG): container finished" podID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerID="4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6" exitCode=0
Nov 03 23:40:19.178802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.178631    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"
Nov 03 23:40:19.178802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.178663    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerDied","Data":"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"}
Nov 03 23:40:19.178802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.178720    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927" event={"ID":"23f94534-eaaf-4282-86c7-1a9844db61fe","Type":"ContainerDied","Data":"af342f1d7c302401f57cf22271f216b1c2c26a8d51d01392b9517a94a1bbafc4"}
Nov 03 23:40:19.178802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.178748    6430 scope.go:122] "RemoveContainer" containerID="4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"
Nov 03 23:40:19.179361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.179342    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:19.180756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.180738    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:19.189419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.189374    6430 scope.go:122] "RemoveContainer" containerID="bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"
Nov 03 23:40:19.198024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.198001    6430 scope.go:122] "RemoveContainer" containerID="4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"
Nov 03 23:40:19.198325 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:19.198304    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6\": container with ID starting with 4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6 not found: ID does not exist" containerID="4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"
Nov 03 23:40:19.198498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.198334    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6"} err="failed to get container status \"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6\": rpc error: code = NotFound desc = could not find container \"4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6\": container with ID starting with 4e8eb72ef89e79e6e275405a2633541c10a4d59b456d7d52e09b1b4158711eb6 not found: ID does not exist"
Nov 03 23:40:19.198498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.198351    6430 scope.go:122] "RemoveContainer" containerID="bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"
Nov 03 23:40:19.198612 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:19.198586    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239\": container with ID starting with bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239 not found: ID does not exist" containerID="bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"
Nov 03 23:40:19.198612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.198603    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239"} err="failed to get container status \"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239\": rpc error: code = NotFound desc = could not find container \"bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239\": container with ID starting with bc947cab4ede19d4d9799da2c9ba15cfbfecf7de9565d74d6279c08b76b75239 not found: ID does not exist"
Nov 03 23:40:19.276414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.275441    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:40:19.278862 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.278825    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d3157179-gw-openshift-default-679b585f8f-v4927"]
Nov 03 23:40:19.351474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.350937    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 23:40:19.351474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:19.351018    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 23:40:20.271597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:20.265505    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" path="/var/lib/kubelet/pods/23f94534-eaaf-4282-86c7-1a9844db61fe/volumes"
Nov 03 23:40:20.451857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:20.435705    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:40:20.451857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:20.443466    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:40:20.464102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:20.464057    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 03 23:40:22.172283 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.172254    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 03 23:40:22.387529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.387491    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 03 23:40:22.387942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.387916    6430 cpu_manager.go:414] "containerMap: removing container" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.387942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.387933    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.388139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.387948    6430 cpu_manager.go:414] "containerMap: removing container" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.388139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.387957    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.388139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.388097    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.388139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.388110    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f94534-eaaf-4282-86c7-1a9844db61fe" containerName="istio-proxy"
Nov 03 23:40:22.391545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.391523    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.432506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.428365    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"default-dockercfg-6wx2m\""
Nov 03 23:40:22.432506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.428552    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"simulator-w4-80f8ba-kserve-self-signed-certs\""
Nov 03 23:40:22.432506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.428623    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"kube-root-ca.crt\""
Nov 03 23:40:22.432736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.432601    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"openshift-service-ca.crt\""
Nov 03 23:40:22.524412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.523299    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 03 23:40:22.558203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558161    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.558372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558228    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.558372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558315    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49m5g\" (UniqueName: \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.558372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558341    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.558372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558368    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.558521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.558419    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659166    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659263    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659349    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49m5g\" (UniqueName: \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659381    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659457    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659455    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659625    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.659975 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.659940    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.661668 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.661625    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.661871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.661850    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.680463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.680374    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49m5g\" (UniqueName: \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\") pod \"simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") " pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:22.703763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:22.703632    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:23.079620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.079595    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 03 23:40:23.081750 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:23.081722    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46bf7f64_b200_40bd_8373_05fbbc3dc847.slice/crio-e8a2ab49b5128611c2e8856fdbc0ca2e91dbbcb4fe58d10771daf4aaf53284b4 WatchSource:0}: Error finding container e8a2ab49b5128611c2e8856fdbc0ca2e91dbbcb4fe58d10771daf4aaf53284b4: Status 404 returned error can't find the container with id e8a2ab49b5128611c2e8856fdbc0ca2e91dbbcb4fe58d10771daf4aaf53284b4
Nov 03 23:40:23.206292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.206232    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" event={"ID":"46bf7f64-b200-40bd-8373-05fbbc3dc847","Type":"ContainerStarted","Data":"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"}
Nov 03 23:40:23.206292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.206277    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" event={"ID":"46bf7f64-b200-40bd-8373-05fbbc3dc847","Type":"ContainerStarted","Data":"e8a2ab49b5128611c2e8856fdbc0ca2e91dbbcb4fe58d10771daf4aaf53284b4"}
Nov 03 23:40:23.239680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.239609    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podStartSLOduration=1.239588825 podStartE2EDuration="1.239588825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:23.238081003 +0000 UTC m=+639.129142938" watchObservedRunningTime="2026-11-03 23:40:23.239588825 +0000 UTC m=+639.130650761"
Nov 03 23:40:23.614298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.614196    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 03 23:40:23.636426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.635590    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 03 23:40:23.636426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.635732    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.645613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.645584    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-c60893-serving-cert\""
Nov 03 23:40:23.725579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.725483    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:23.783031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.782996    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8bf\" (UniqueName: \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.783200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.783075    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.783200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.783109    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.783200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.783126    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.884817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.884720    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.884817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.884772    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.885015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.884879    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8bf\" (UniqueName: \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.885015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.884932    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.885609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.885582    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.887888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.887863    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.889759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.889735    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.898778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.898755    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8bf\" (UniqueName: \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\") pod \"maas-api-e2e-worker-w3-c60893-698545b546-g26bh\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:23.951574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:23.951535    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:24.256710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:24.256686    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 03 23:40:24.259255 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:24.259211    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7078d1f9_107d_44cc_b2d9_5b064ed72fc4.slice/crio-12e3377a88e4f40d8240bc5a7fb181f5bd3c84742d54d6303008c5528998e4ca WatchSource:0}: Error finding container 12e3377a88e4f40d8240bc5a7fb181f5bd3c84742d54d6303008c5528998e4ca: Status 404 returned error can't find the container with id 12e3377a88e4f40d8240bc5a7fb181f5bd3c84742d54d6303008c5528998e4ca
Nov 03 23:40:25.221124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:25.221080    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" event={"ID":"7078d1f9-107d-44cc-b2d9-5b064ed72fc4","Type":"ContainerStarted","Data":"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"}
Nov 03 23:40:25.221336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:25.221187    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" event={"ID":"7078d1f9-107d-44cc-b2d9-5b064ed72fc4","Type":"ContainerStarted","Data":"12e3377a88e4f40d8240bc5a7fb181f5bd3c84742d54d6303008c5528998e4ca"}
Nov 03 23:40:25.221336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:25.221280    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:25.256348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:25.256232    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" podStartSLOduration=1.840840296 podStartE2EDuration="2.256217751s" totalImagesPullingTime="415.377455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:25.255931303 +0000 UTC m=+641.146993241" watchObservedRunningTime="2026-11-03 23:40:25.256217751 +0000 UTC m=+641.147279689"
Nov 03 23:40:26.052782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:26.052739    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:28.238822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.238782    6430 generic.go:356] "Generic (PLEG): container finished" podID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerID="b828cb530ef173a45db5398a5088d3dc061579fe2084445e6b7a650e46d885fc" exitCode=0
Nov 03 23:40:28.239268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.238851    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerDied","Data":"b828cb530ef173a45db5398a5088d3dc061579fe2084445e6b7a650e46d885fc"}
Nov 03 23:40:28.239317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.239269    6430 scope.go:122] "RemoveContainer" containerID="b828cb530ef173a45db5398a5088d3dc061579fe2084445e6b7a650e46d885fc"
Nov 03 23:40:28.240659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.240630    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:28.240743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.240710    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:28.240780 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:28.240741    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:29.246583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:29.246546    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerStarted","Data":"98aa226e2bf2ce341174c79676badd9dd1b6efcfb6c8f3f5fb14a70afddf645a"}
Nov 03 23:40:30.001966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:30.001923    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:30.002175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:30.002138    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 23:40:30.002254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:30.002209    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 23:40:30.830773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:30.820457    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:31.001785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:31.001748    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 23:40:31.001972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:31.001805    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 23:40:31.237263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:31.237173    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 03 23:40:32.002644 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:32.002603    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 23:40:32.003049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:32.002683    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 23:40:32.716414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:32.713774    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:32.742737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:32.742704    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:33.002025 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:33.001927    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 23:40:33.002025 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:33.001990    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 23:40:33.303941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:33.303920    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:40:34.002065 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:34.002024    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 23:40:34.002274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:34.002100    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 23:40:34.833900 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:40:34.833861    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:40:35.006232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:35.006202    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:35.006607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:35.006588    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:35.007326 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:35.007307    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 03 23:40:39.729287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:39.729250    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:42.559901 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.559866    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 03 23:40:42.564580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.564553    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.580435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.580375    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 03 23:40:42.580927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.580909    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"premium-w4-80f8ba-kserve-self-signed-certs\""
Nov 03 23:40:42.640297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640253    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.640488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640373    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.640488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640442    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmgt\" (UniqueName: \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.640488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.640609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640545    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.640665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.640642    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746040    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746138    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746170    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmgt\" (UniqueName: \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746204    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746253    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746305    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746787    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.746937    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.747990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.747534    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.752962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.752929    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.753905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.753883    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.768823 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.768801    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmgt\" (UniqueName: \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\") pod \"premium-w4-80f8ba-kserve-788fdd7d99-jbjp7\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") " pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:42.876419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:42.876314    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:43.175000 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:43.174972    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 03 23:40:43.177194 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:43.177168    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf47b886_ddb6_497a_b569_85fed4ca9c36.slice/crio-1c493aed4fcf5a9ee33318870e08cbb2a1aef5c08cf274b5ea3f61ac922e331c WatchSource:0}: Error finding container 1c493aed4fcf5a9ee33318870e08cbb2a1aef5c08cf274b5ea3f61ac922e331c: Status 404 returned error can't find the container with id 1c493aed4fcf5a9ee33318870e08cbb2a1aef5c08cf274b5ea3f61ac922e331c
Nov 03 23:40:43.363451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:43.363408    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" event={"ID":"df47b886-ddb6-497a-b569-85fed4ca9c36","Type":"ContainerStarted","Data":"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"}
Nov 03 23:40:43.363640 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:43.363466    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" event={"ID":"df47b886-ddb6-497a-b569-85fed4ca9c36","Type":"ContainerStarted","Data":"1c493aed4fcf5a9ee33318870e08cbb2a1aef5c08cf274b5ea3f61ac922e331c"}
Nov 03 23:40:43.397797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:43.397725    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podStartSLOduration=1.397703227 podStartE2EDuration="1.397703227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:43.392825606 +0000 UTC m=+659.283887548" watchObservedRunningTime="2026-11-03 23:40:43.397703227 +0000 UTC m=+659.288765173"
Nov 03 23:40:43.877013 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:43.876977    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:46.261763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.261723    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 03 23:40:46.267266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.267239    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.272230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.272207    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"kube-root-ca.crt\""
Nov 03 23:40:46.272530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.272513    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"openshift-service-ca.crt\""
Nov 03 23:40:46.275163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.275146    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"default-dockercfg-fjmq9\""
Nov 03 23:40:46.275248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.275183    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"simulator-w3-c60893-kserve-self-signed-certs\""
Nov 03 23:40:46.286481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.286451    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.286728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.286495    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.287462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.287436    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26l9r\" (UniqueName: \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.287615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.287580    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.287744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.287648    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.287744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.287716    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.289560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.289536    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 03 23:40:46.388959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.388924    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.388959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.388959    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l9r\" (UniqueName: \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.388991    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389032    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389090    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389205    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389545    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.389633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.389618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.391437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.391411    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.391582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.391566    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.401844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.401822    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26l9r\" (UniqueName: \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\") pod \"simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") " pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.579778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.579740    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:46.862081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:46.862050    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 03 23:40:46.863311 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:46.863285    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d463f76_9ae4_4b76_848d_2ace5fecfb94.slice/crio-a6015c9dc9b575ec7920a79a744014c56841532de9152a1a2c5b99f95a4d0e99 WatchSource:0}: Error finding container a6015c9dc9b575ec7920a79a744014c56841532de9152a1a2c5b99f95a4d0e99: Status 404 returned error can't find the container with id a6015c9dc9b575ec7920a79a744014c56841532de9152a1a2c5b99f95a4d0e99
Nov 03 23:40:47.385463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:47.385427    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" event={"ID":"6d463f76-9ae4-4b76-848d-2ace5fecfb94","Type":"ContainerStarted","Data":"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"}
Nov 03 23:40:47.385463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:47.385469    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" event={"ID":"6d463f76-9ae4-4b76-848d-2ace5fecfb94","Type":"ContainerStarted","Data":"a6015c9dc9b575ec7920a79a744014c56841532de9152a1a2c5b99f95a4d0e99"}
Nov 03 23:40:47.407837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:47.407787    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podStartSLOduration=1.407774396 podStartE2EDuration="1.407774396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:47.406009557 +0000 UTC m=+663.297071512" watchObservedRunningTime="2026-11-03 23:40:47.407774396 +0000 UTC m=+663.298836330"
Nov 03 23:40:47.580649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:47.580612    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:48.832345 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:48.832308    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:40:49.580085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:49.579903    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:58220->10.134.0.42:15021: read: connection reset by peer" start-of-body=
Nov 03 23:40:49.580085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:49.579990    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:58220->10.134.0.42:15021: read: connection reset by peer"
Nov 03 23:40:50.409777 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.409737    6430 generic.go:356] "Generic (PLEG): container finished" podID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerID="266c03d8800ce3d9c0f88022f87141afa4e18ec23044ab9cbec79bf6f5a531db" exitCode=0
Nov 03 23:40:50.410198 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.409806    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerDied","Data":"266c03d8800ce3d9c0f88022f87141afa4e18ec23044ab9cbec79bf6f5a531db"}
Nov 03 23:40:50.410243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.410228    6430 scope.go:122] "RemoveContainer" containerID="266c03d8800ce3d9c0f88022f87141afa4e18ec23044ab9cbec79bf6f5a531db"
Nov 03 23:40:50.411557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.411527    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:50.411679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.411613    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:50.411742 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:50.411687    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:40:51.437510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:51.437477    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerStarted","Data":"dfb0101c8e917b14dcd485629455fb7583c90a25df5c8c3f0100ec7b922bc41f"}
Nov 03 23:40:52.051317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:52.051278    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:52.051576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:52.051549    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 23:40:52.051660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:52.051608    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 23:40:52.877219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:52.877176    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:52.890336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:52.890302    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:53.051067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:53.051038    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 23:40:53.051252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:53.051093    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 23:40:53.459721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:53.459693    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:40:54.051906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:54.051870    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 23:40:54.052315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:54.051924    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 23:40:55.063001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:55.062699    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 23:40:55.063001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:55.062780    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 23:40:56.051768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:56.051730    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 23:40:56.051965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:56.051812    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 23:40:56.580120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:56.580081    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:56.593827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:56.593794    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:57.055552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:57.055519    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:57.055794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:57.055774    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:57.056355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:57.056336    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 03 23:40:57.488173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:57.488088    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:40:58.898892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.898862    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 03 23:40:58.903445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.903417    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:58.909949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.909921    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"default-dockercfg-6tjp2\""
Nov 03 23:40:58.911923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.911903    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"simulator-w1-5cb349-kserve-self-signed-certs\""
Nov 03 23:40:58.912299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.912283    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"openshift-service-ca.crt\""
Nov 03 23:40:58.912377 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.912309    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"kube-root-ca.crt\""
Nov 03 23:40:58.928256 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:58.928225    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 03 23:40:59.034544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034508    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.034544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034544    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.034794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034573    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.034794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034745    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.034874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034851    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.034908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.034889    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfpn\" (UniqueName: \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136083    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136153    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfpn\" (UniqueName: \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136212    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136247    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136301    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136422    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136642    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136819    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.136954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.136864    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.138814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.138794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.139021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.138999    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.150021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.149966    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfpn\" (UniqueName: \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\") pod \"simulator-w1-5cb349-kserve-69954974f6-9f8lk\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") " pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.219767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.219735    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:40:59.515215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:40:59.515178    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 03 23:40:59.517914 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:40:59.517885    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod689807c7_203c_4dec_82fa_a2a8925be4ae.slice/crio-e99e6f34f66c5dc87f9dba4e1afd4f67578b66ecc5704033a676424e24db44d4 WatchSource:0}: Error finding container e99e6f34f66c5dc87f9dba4e1afd4f67578b66ecc5704033a676424e24db44d4: Status 404 returned error can't find the container with id e99e6f34f66c5dc87f9dba4e1afd4f67578b66ecc5704033a676424e24db44d4
Nov 03 23:41:00.324641 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:00.324604    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:41:00.492582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:00.492544    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" event={"ID":"689807c7-203c-4dec-82fa-a2a8925be4ae","Type":"ContainerStarted","Data":"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"}
Nov 03 23:41:00.492763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:00.492596    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" event={"ID":"689807c7-203c-4dec-82fa-a2a8925be4ae","Type":"ContainerStarted","Data":"e99e6f34f66c5dc87f9dba4e1afd4f67578b66ecc5704033a676424e24db44d4"}
Nov 03 23:41:00.519572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:00.519513    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podStartSLOduration=2.519494602 podStartE2EDuration="2.519494602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:00.518589404 +0000 UTC m=+676.409651340" watchObservedRunningTime="2026-11-03 23:41:00.519494602 +0000 UTC m=+676.410556537"
Nov 03 23:41:01.220673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:01.220642    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:41:04.831040 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:41:04.830995    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:41:09.219949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:09.219917    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:41:09.232473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:09.232449    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:41:09.588338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:09.588311    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:41:11.078294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:11.078252    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:41:30.183126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.183087    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 03 23:41:30.187236 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.187210    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.191962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.191937    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c60893-openshift-default-dockercfg-hwjgp\""
Nov 03 23:41:30.199463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.199439    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 03 23:41:30.239124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239087    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239198    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239238    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239283    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239357    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239489    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239653    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvkm\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.239770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.239754    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340608 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvkm\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340630    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340674    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340723    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340758    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.340799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340796    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340827    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340868    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.340905    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.341228    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341384 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.341355    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.341355    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.341459    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.341743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.341722    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.343220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.343202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.343665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.343648    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.354055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.354024    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.354157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.354112    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvkm\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.503068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.502986    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:30.787093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.787053    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 03 23:41:30.789584 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:41:30.789550    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa581b61_8d00_4f7d_81a2_4f2cf4b2fa66.slice/crio-8ee560fcec782485924015e23b6a22dee0eaec88751d068bc0b6a5bd39ee5d27 WatchSource:0}: Error finding container 8ee560fcec782485924015e23b6a22dee0eaec88751d068bc0b6a5bd39ee5d27: Status 404 returned error can't find the container with id 8ee560fcec782485924015e23b6a22dee0eaec88751d068bc0b6a5bd39ee5d27
Nov 03 23:41:30.792093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.792050    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:30.792191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.792139    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:30.792232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:30.792186    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:31.642473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:31.642431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:41:31.688831 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:31.688800    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" event={"ID":"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66","Type":"ContainerStarted","Data":"e22812dcd597ee3c770d2489264d551fa618aec182499b03c9f8a084ff49ab5e"}
Nov 03 23:41:31.689041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:31.688852    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" event={"ID":"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66","Type":"ContainerStarted","Data":"8ee560fcec782485924015e23b6a22dee0eaec88751d068bc0b6a5bd39ee5d27"}
Nov 03 23:41:31.732643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:31.732571    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podStartSLOduration=1.732547147 podStartE2EDuration="1.732547147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:31.719710973 +0000 UTC m=+707.610772913" watchObservedRunningTime="2026-11-03 23:41:31.732547147 +0000 UTC m=+707.623609086"
Nov 03 23:41:32.503710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:32.503654    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:32.505459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:32.505428    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 23:41:32.505598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:32.505481    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 23:41:33.504191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:33.504157    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 23:41:33.504636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:33.504226    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 23:41:34.504417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:34.504360    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 23:41:34.504850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:34.504466    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 23:41:34.856414 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:41:34.856336    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:41:35.503632 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:35.503593    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 23:41:35.503821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:35.503651    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 23:41:36.504132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:36.504096    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 23:41:36.504569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:36.504164    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 23:41:38.408096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:38.408049    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:38.408641 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:38.408545    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:38.409497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:38.409475    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 03 23:41:43.970271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:43.970230    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 03 23:41:43.974639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:43.974615    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:43.979558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:43.979531    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f6f292-openshift-default-dockercfg-b884k\""
Nov 03 23:41:43.985972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:43.985944    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 03 23:41:44.069721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.069673    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.069927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.069730    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52fd\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.069927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.069822    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.069927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.069850    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.069927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.069879    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.070087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.070049    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.070170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.070149    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.070231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.070220    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.070266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.070243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.170933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.170894    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.170933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.170940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52fd\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.170962    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171084    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171118    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171193    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171285    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171361    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171426    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171428    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171735 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.171876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.171741    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.172103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.172083    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.173544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.173522    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.173899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.173879    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.184676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.184644    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52fd\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.184810 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.184788    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.295507 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.295414    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f6f292-openshift-default-dockercfg-b884k\""
Nov 03 23:41:44.302231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.302203    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:44.593886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.593862    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 03 23:41:44.595660 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:41:44.595633    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7286d5c3_0565_45a2_bc50_20d67023ce69.slice/crio-b51bae8b161c35610d6efccf5bde7dcc7303d51fd753dd8f7453f125ae9251e1 WatchSource:0}: Error finding container b51bae8b161c35610d6efccf5bde7dcc7303d51fd753dd8f7453f125ae9251e1: Status 404 returned error can't find the container with id b51bae8b161c35610d6efccf5bde7dcc7303d51fd753dd8f7453f125ae9251e1
Nov 03 23:41:44.597944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.597916    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:44.598012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.597976    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:44.598012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.598003    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:41:44.776017 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.775979    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" event={"ID":"7286d5c3-0565-45a2-bc50-20d67023ce69","Type":"ContainerStarted","Data":"23a95723a45269ae4293c85798db374f1383aa98b03a7c5fd37fe9564a64349f"}
Nov 03 23:41:44.776191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.776042    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" event={"ID":"7286d5c3-0565-45a2-bc50-20d67023ce69","Type":"ContainerStarted","Data":"b51bae8b161c35610d6efccf5bde7dcc7303d51fd753dd8f7453f125ae9251e1"}
Nov 03 23:41:44.800956 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:44.800901    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podStartSLOduration=1.800887442 podStartE2EDuration="1.800887442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:44.800744684 +0000 UTC m=+720.691806619" watchObservedRunningTime="2026-11-03 23:41:44.800887442 +0000 UTC m=+720.691949378"
Nov 03 23:41:45.304026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:45.303985    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:45.305717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:45.305689    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:45.305828 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:45.305741    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:46.302817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:46.302782    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:46.303135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:46.302834    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:47.303207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:47.303168    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:47.303723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:47.303223    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:48.303254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:48.303214    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:48.303727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:48.303287    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:49.303647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:49.303614    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:49.304044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:49.303668    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:50.303800 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:50.303769    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 23:41:50.304182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:50.303818    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 23:41:51.399795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:51.399755    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:51.400407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:51.400366    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:51.401976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:51.401952    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 03 23:41:52.233955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:41:52.233923    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:42:00.394582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:00.394483    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:42:00.394582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:00.394544    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:42:04.904380 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:42:04.904336    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:42:12.427809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:12.427769    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:42:15.404585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:15.404535    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:42:15.405071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:15.404614    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:42:17.844155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.844111    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 03 23:42:17.849924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.849899    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.863926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.863894    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 03 23:42:17.935258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935221    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935261    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935294    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hb47\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935450    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935486    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935507    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935709 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935599    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935729    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.935790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.935747    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:17.946106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.946070    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:42:17.949825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.949802    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:17.958501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.958475    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"e2e-conflict-a-86358670-kserve-self-signed-certs\""
Nov 03 23:42:17.959958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:17.959928    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041131    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041263    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041296    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041332    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hb47\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041418    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041450    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041484    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.048410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.041513    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.049015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.048794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.065653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.049070    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.065653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.049744    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.065653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.050073    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.065653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.050238    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.065653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.051693    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.093218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.093185    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hb47\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.093474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.093403    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.099727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.099646    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ppkrb\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.143004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.142968    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.143178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.143012    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.143178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.143157    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.143270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.143184    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.143330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.143293    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn4t\" (UniqueName: \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.143435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.143419    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.169060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.169015    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.247939    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.247986    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn4t\" (UniqueName: \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248068    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248104    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248126    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248553    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.248842    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.249314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.249205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.251347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.251313    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.252750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.252460    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.264422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.263896    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn4t\" (UniqueName: \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\") pod \"e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") " pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.460726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.460700    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 03 23:42:18.467505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.467464    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:18.467605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.467535    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:18.467605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.467565    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:18.563935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.563892    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:18.850883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:18.850858    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:42:18.853007 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:42:18.852978    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96963007_afa9_4dfa_80a6_1ea9df272a6f.slice/crio-9c253b47caff7c46e81c25c53e1cafaf8e164908a0d2c03d3afe0a9d886313a9 WatchSource:0}: Error finding container 9c253b47caff7c46e81c25c53e1cafaf8e164908a0d2c03d3afe0a9d886313a9: Status 404 returned error can't find the container with id 9c253b47caff7c46e81c25c53e1cafaf8e164908a0d2c03d3afe0a9d886313a9
Nov 03 23:42:19.019418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.019259    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" event={"ID":"96963007-afa9-4dfa-80a6-1ea9df272a6f","Type":"ContainerStarted","Data":"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"}
Nov 03 23:42:19.019418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.019333    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" event={"ID":"96963007-afa9-4dfa-80a6-1ea9df272a6f","Type":"ContainerStarted","Data":"9c253b47caff7c46e81c25c53e1cafaf8e164908a0d2c03d3afe0a9d886313a9"}
Nov 03 23:42:19.020929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.020894    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" event={"ID":"b82b45da-12c4-426b-aa07-955f9d903f8d","Type":"ContainerStarted","Data":"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"}
Nov 03 23:42:19.021082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.020959    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" event={"ID":"b82b45da-12c4-426b-aa07-955f9d903f8d","Type":"ContainerStarted","Data":"f754cee307489bb15c3ae3d5416dc97fb3ab2b6e6f7d214e3c4e38dc63e28537"}
Nov 03 23:42:19.043540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.043478    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" podStartSLOduration=2.043461143 podStartE2EDuration="2.043461143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:19.041231059 +0000 UTC m=+754.932292994" watchObservedRunningTime="2026-11-03 23:42:19.043461143 +0000 UTC m=+754.934523072"
Nov 03 23:42:19.074317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.074234    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podStartSLOduration=2.074210323 podStartE2EDuration="2.074210323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:19.071975464 +0000 UTC m=+754.963037413" watchObservedRunningTime="2026-11-03 23:42:19.074210323 +0000 UTC m=+754.965272260"
Nov 03 23:42:19.173476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.169276    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:19.173476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.172864    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:19.173476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.172923    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:19.564600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:19.564561    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:20.176551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:20.176514    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:20.177155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:20.177121    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:21.170137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:21.170099    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:21.170352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:21.170165    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:22.170414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:22.169639    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:22.170414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:22.169705    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:23.169927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:23.169885    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:23.170274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:23.170227    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:23.325534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:23.325486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:42:24.169523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:24.169483    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:24.169707 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:24.169538    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:25.169744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:25.169702    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:25.170168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:25.169773    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:26.198344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:26.198303    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 23:42:26.198965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:26.198930    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 23:42:27.457322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:27.457281    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:27.458147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:27.458114    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:27.458880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:27.458856    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 03 23:42:28.564698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:28.564649    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:28.578335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:28.578306    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:29.241431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:29.241373    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:42:30.887094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:30.887055    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:42:35.099270 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:42:35.099222    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:42:41.740323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.740264    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 03 23:42:41.743969 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.743948    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.752813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.752785    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"kube-root-ca.crt\""
Nov 03 23:42:41.755275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.755152    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"openshift-service-ca.crt\""
Nov 03 23:42:41.755275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.755249    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"default-dockercfg-5kdv6\""
Nov 03 23:42:41.755698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.755585    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"distinct-2-w2-5738a9-kserve-self-signed-certs\""
Nov 03 23:42:41.758077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.757954    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 03 23:42:41.806227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806187    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.806376 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806304    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.806474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806433    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjkn\" (UniqueName: \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.806474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806470    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.806551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806500    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.806551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.806517    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908081    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjkn\" (UniqueName: \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908152    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908198    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908222    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908298    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.908874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.909361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.908943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.910402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.910359    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.910845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.910822    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.930343 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.928679    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.930343 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.928952    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:41.931183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:41.931153    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjkn\" (UniqueName: \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\") pod \"distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:42.058272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:42.058244    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:42.334360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:42.334334    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 03 23:42:42.336683 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:42:42.336655    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d7b1139_91b7_44e9_a78f_e8f4d46e83ed.slice/crio-7e5bcc1e1a5e20b033727e166f560da171212e84e026b5903d15f7e8b4515bae WatchSource:0}: Error finding container 7e5bcc1e1a5e20b033727e166f560da171212e84e026b5903d15f7e8b4515bae: Status 404 returned error can't find the container with id 7e5bcc1e1a5e20b033727e166f560da171212e84e026b5903d15f7e8b4515bae
Nov 03 23:42:42.451212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:42.451176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" event={"ID":"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed","Type":"ContainerStarted","Data":"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"}
Nov 03 23:42:42.451403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:42.451228    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" event={"ID":"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed","Type":"ContainerStarted","Data":"7e5bcc1e1a5e20b033727e166f560da171212e84e026b5903d15f7e8b4515bae"}
Nov 03 23:42:42.473619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:42.473566    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podStartSLOduration=1.473553244 podStartE2EDuration="1.473553244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:42.471871818 +0000 UTC m=+778.362933778" watchObservedRunningTime="2026-11-03 23:42:42.473553244 +0000 UTC m=+778.364615179"
Nov 03 23:42:43.058625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:43.058590    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:51.711249 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:51.711214    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:42:52.059365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.059328    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:52.071584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.071552    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:52.575686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.575656    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:42:52.649812 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.649776    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:42:52.655114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.655089    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.667829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.667804    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:42:52.792286 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792298    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792448    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792484    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792577    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792643    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76djb\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792666    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792705    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.792995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.792821    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894436    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894483    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894535    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894557    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894607    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894646    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76djb\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894675    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894728    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.894848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.894800    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.895335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.895302    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.895615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.895589    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.895704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.895654    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.895911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.895886    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.896077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.895938    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.897694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.897642    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.897966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.897942    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.908381 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.908361    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.908479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.908440    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76djb\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:52.975303 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:52.975255    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:53.319411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.316116    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:42:53.359207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.352737    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:53.359207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.352826    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:53.359207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.352867    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:42:53.564415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.564355    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" event={"ID":"c3b487d7-f998-4f38-bbe4-54aba5fccfc1","Type":"ContainerStarted","Data":"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"}
Nov 03 23:42:53.564809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.564700    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" event={"ID":"c3b487d7-f998-4f38-bbe4-54aba5fccfc1","Type":"ContainerStarted","Data":"3dabd35f8f25d567109d0c19e55f7ec38a09058292d15632945ef8e7d21859df"}
Nov 03 23:42:53.591196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.591100    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podStartSLOduration=1.591086594 podStartE2EDuration="1.591086594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:53.589533251 +0000 UTC m=+789.480595235" watchObservedRunningTime="2026-11-03 23:42:53.591086594 +0000 UTC m=+789.482148529"
Nov 03 23:42:53.975813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:53.975725    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:42:54.977160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:54.977128    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:54.977631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:54.977180    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:42:55.975912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:55.975878    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:55.976097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:55.975932    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:42:56.976414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:56.976352    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:56.976809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:56.976437    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:42:57.976499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:57.976462    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:57.976904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:57.976532    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:42:58.976305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:58.976269    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:58.976523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:58.976332    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:42:59.979902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:59.979859    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 23:42:59.980524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:42:59.980486    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 23:43:01.856905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:01.856870    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:43:01.857365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:01.857245    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:43:01.858162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:01.858143    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:43:04.043611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.043552    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:43:04.047637 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.047616    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.059499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.059448    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:43:04.132301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132266    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132313    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132344    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132560    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132649    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgw86\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132742    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132765    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.132821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.132794    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.233707 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgw86\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.233888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.233888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233783    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.233888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233813    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.233888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233847    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.233937    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234030    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234051    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234225    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234246    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234326    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234422    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.234799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.234774    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.236273 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.236244    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.236383 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.236365    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.247321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.247286    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.247816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.247794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgw86\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.365201 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.365169    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:04.750966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.750938    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:43:04.752836 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:43:04.752810    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d70def_e6b7_4ed1_a7cf_0f6413475b23.slice/crio-ddd4db1edbb6dd087a41ba7e89db35296626df41303ddec76f3d31b00d23d4e5 WatchSource:0}: Error finding container ddd4db1edbb6dd087a41ba7e89db35296626df41303ddec76f3d31b00d23d4e5: Status 404 returned error can't find the container with id ddd4db1edbb6dd087a41ba7e89db35296626df41303ddec76f3d31b00d23d4e5
Nov 03 23:43:04.755260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.755227    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:04.755366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.755293    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:04.755366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:04.755329    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:04.832586 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:43:04.832546    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:43:05.736847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:05.736803    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" event={"ID":"38d70def-e6b7-4ed1-a7cf-0f6413475b23","Type":"ContainerStarted","Data":"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"}
Nov 03 23:43:05.737259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:05.736875    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" event={"ID":"38d70def-e6b7-4ed1-a7cf-0f6413475b23","Type":"ContainerStarted","Data":"ddd4db1edbb6dd087a41ba7e89db35296626df41303ddec76f3d31b00d23d4e5"}
Nov 03 23:43:05.765202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:05.765157    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podStartSLOduration=1.765143857 podStartE2EDuration="1.765143857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:05.76378516 +0000 UTC m=+801.654847106" watchObservedRunningTime="2026-11-03 23:43:05.765143857 +0000 UTC m=+801.656205791"
Nov 03 23:43:06.365482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:06.365439    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:06.366822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:06.366798    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:06.366933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:06.366846    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:07.365978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:07.365941    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:07.366413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:07.365993    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:08.366338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:08.366292    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:08.366788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:08.366354    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:09.365931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:09.365890    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:09.366121 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:09.365954    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:10.365748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:10.365700    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:10.366260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:10.365779    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:11.366910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:11.366547    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 23:43:11.366910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:11.366609    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 23:43:12.370153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:12.370119    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:12.370586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:12.370426    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:12.371049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:12.371031    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:43:12.986704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:12.986670    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:43:19.109103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.109070    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 03 23:43:19.114203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.114181    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.126493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.126468    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 03 23:43:19.203399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203355    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.203558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203429    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.203627 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203598    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.203757 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203720    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.203892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203821    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.203892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203862    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.204013 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203955    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.204013 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.203982    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq72j\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.204124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.204017    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.304936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.304901    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.304954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.304993    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305040    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305068    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305151    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305178    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq72j\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305224    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305425    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305504    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305901    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.305993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.305925    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.306347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.306325    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.307604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.307581    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.307708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.307650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.320763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.320733    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.320968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.320946    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq72j\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.459719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.459629    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:19.759365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.759339    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 03 23:43:19.761247 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:43:19.761217    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ba74fe_5fae_4505_ba57_83a44c9b884d.slice/crio-f75d8e754f7165691f7e1656329573a283e5808caff24bca7f18deae5016571b WatchSource:0}: Error finding container f75d8e754f7165691f7e1656329573a283e5808caff24bca7f18deae5016571b: Status 404 returned error can't find the container with id f75d8e754f7165691f7e1656329573a283e5808caff24bca7f18deae5016571b
Nov 03 23:43:19.763784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.763752    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:19.763842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.763816    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:19.763892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.763842    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:19.900674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.900640    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" event={"ID":"a7ba74fe-5fae-4505-ba57-83a44c9b884d","Type":"ContainerStarted","Data":"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"}
Nov 03 23:43:19.900868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.900688    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" event={"ID":"a7ba74fe-5fae-4505-ba57-83a44c9b884d","Type":"ContainerStarted","Data":"f75d8e754f7165691f7e1656329573a283e5808caff24bca7f18deae5016571b"}
Nov 03 23:43:19.929110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:19.929042    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podStartSLOduration=0.929023314 podStartE2EDuration="929.023314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:19.92701706 +0000 UTC m=+815.818079028" watchObservedRunningTime="2026-11-03 23:43:19.929023314 +0000 UTC m=+815.820085251"
Nov 03 23:43:20.474410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:20.466901    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:20.474410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:20.468706    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:20.474410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:20.468764    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:21.473600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:21.473557    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:43:21.479931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:21.479511    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:21.479931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:21.479570    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:22.460049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.460014    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:22.460261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.460067    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:22.871116 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.871072    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:43:22.874652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.874631    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899454    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6z7\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899656 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899656 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899628    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899655    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899701    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899882 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899867    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899928    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.899996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.899953    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:22.942446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:22.929009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:43:23.001496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001510    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6z7\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001571    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001598    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001634    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001660    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001749    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001810    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.001943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.001834    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.002114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.002027    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.002114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.002085    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.002194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.002171    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.002229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.002210    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.002450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.002424    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.004124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.004099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.004523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.004505    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.033167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.033063    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.035971 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.035939    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6z7\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.188603 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.188514    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:23.460523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.460428    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:23.460523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.460499    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:23.532794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.532710    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:43:23.537587 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:43:23.537554    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb070961_7e25_4bb4_ae8b_1953dfffa534.slice/crio-4fec17a6950e2f2a06c0e994c7d986eed6b25f39a543841b5a431f4f205d9c25 WatchSource:0}: Error finding container 4fec17a6950e2f2a06c0e994c7d986eed6b25f39a543841b5a431f4f205d9c25: Status 404 returned error can't find the container with id 4fec17a6950e2f2a06c0e994c7d986eed6b25f39a543841b5a431f4f205d9c25
Nov 03 23:43:23.539990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.539956    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:23.540083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.540025    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:23.540083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.540051    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:23.957638 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.957597    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" event={"ID":"cb070961-7e25-4bb4-ae8b-1953dfffa534","Type":"ContainerStarted","Data":"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"}
Nov 03 23:43:23.958109 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.957677    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" event={"ID":"cb070961-7e25-4bb4-ae8b-1953dfffa534","Type":"ContainerStarted","Data":"4fec17a6950e2f2a06c0e994c7d986eed6b25f39a543841b5a431f4f205d9c25"}
Nov 03 23:43:23.990841 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:23.990788    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podStartSLOduration=1.990773376 podStartE2EDuration="1.990773376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:23.990312875 +0000 UTC m=+819.881374829" watchObservedRunningTime="2026-11-03 23:43:23.990773376 +0000 UTC m=+819.881835311"
Nov 03 23:43:24.203126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:24.203092    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:24.211474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:24.209517    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:24.211474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:24.209581    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:24.462491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:24.460617    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:24.462491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:24.460699    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:25.189720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:25.189687    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:25.190123 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:25.189740    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:25.462466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:25.462055    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:25.462466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:25.462130    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:26.189885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:26.189835    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:26.190437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:26.189947    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:26.460134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:26.460030    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 23:43:26.460134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:26.460102    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 23:43:27.189680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:27.189631    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:27.189892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:27.189728    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:27.464719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:27.464634    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:27.465151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:27.465133    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:27.465825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:27.465803    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 03 23:43:28.188850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:28.188816    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:28.189044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:28.188874    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:29.189316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:29.189285    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:29.189723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:29.189336    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:30.193412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.192869    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:30.193412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.192934    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:30.434405 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.434348    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 03 23:43:30.459781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.458026    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 03 23:43:30.459781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.458152    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.502316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.502275    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.502609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.502554    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.502917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.502895    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.503113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.503077    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmbd\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.503304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.503270    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.503490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.503471    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.503633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.503616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.503973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.503955    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.504132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.504109    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.681415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.674174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.681826 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.681776    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmbd\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.682679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.682658    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690448    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690587    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690647    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690708    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690793    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.695418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.690822    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.721577 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.696050    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.745754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.702217    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.745993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.702500    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.746108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.733281    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.840423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.818943    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.897414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.892710    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmbd\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.972843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.972750    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:30.974809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:30.974740    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:31.137675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.137635    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:31.213483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.203541    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 23:43:31.213483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.203606    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 23:43:31.265703 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.263590    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 23:43:31.265990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.265941    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:31.297739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.297696    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:31.457416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.450571    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:31.627766 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.626710    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:31.739625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:31.738507    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:32.331477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.287164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:43:32.347413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.335465    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:32.347413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.338190    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:43:32.559413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.558657    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:43:32.559413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.559302    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:32.814469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.809501    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:32.814469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.811529    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.828965    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.829098    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.829287    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.829326    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.829359    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:43:32.863412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.829490    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:43:32.890699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:32.888835    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 03 23:43:33.036662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.036614    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:33.037143 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.037111    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:33.037367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.037346    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:43:33.236412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.235115    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 03 23:43:33.324539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.299778    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:43:33.324539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.299853    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.645581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.645532    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:43:33.699745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.649135    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.706736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.653667    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.712414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.707102    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:43:33.714495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.654276    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 23:43:33.835957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.835900    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.907544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.896747    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.981228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.974629    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:33.998354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:33.995476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.075224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.075163    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.219468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.216303    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" event={"ID":"1fb12ba6-dce0-4c74-b25e-073043cbc811","Type":"ContainerStarted","Data":"d2754aa0644dcbfb11ff9173fc1062d24dd2b47c19f7d3e8900b8fb9dddd9c02"}
Nov 03 23:43:34.389605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.389565    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:43:34.389916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.389868    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.515872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.515842    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:43:34.644592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.621240    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:43:34.727454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.707779    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:43:34.727454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.709066    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:43:34.760572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.741311    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:34.862822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.834554    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:43:34.862822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.847084    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:43:34.862822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.841785    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:43:34.930045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.925329    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:43:34.938415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.930508    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:43:34.952344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.944062    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:43:34.976198 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:43:34.975965    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:43:34.980907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.979736    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:34.982554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.982527    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:34.984170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:34.984137    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:43:35.277375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:35.277265    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" event={"ID":"1fb12ba6-dce0-4c74-b25e-073043cbc811","Type":"ContainerStarted","Data":"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"}
Nov 03 23:43:35.310607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:35.310538    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podStartSLOduration=5.310517766 podStartE2EDuration="5.310517766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:35.30800409 +0000 UTC m=+831.199066028" watchObservedRunningTime="2026-11-03 23:43:35.310517766 +0000 UTC m=+831.201579702"
Nov 03 23:43:36.138351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:36.138309    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:36.139794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:36.139760    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:36.139926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:36.139823    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:36.532059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:36.531952    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main" containerID="cri-o://a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8" gracePeriod=45
Nov 03 23:43:36.904098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:36.904066    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:43:37.002102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002067    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002114    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002164    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002223    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002250    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002298    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\" (UniqueName: \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\") pod \"96963007-afa9-4dfa-80a6-1ea9df272a6f\" (UID: \"96963007-afa9-4dfa-80a6-1ea9df272a6f\") "
Nov 03 23:43:37.002510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002448    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:37.002625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002523    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:37.002625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002552    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:37.002793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002776    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.002838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002796    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.002838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.002805    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.004450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.004421    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:43:37.004746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.004728    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:43:37.004818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.004749    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t" pod "96963007-afa9-4dfa-80a6-1ea9df272a6f" (UID: "96963007-afa9-4dfa-80a6-1ea9df272a6f"). InnerVolumeSpecName "kube-api-access-qcn4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:43:37.104167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.104130    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcn4t\" (UniqueName: \"kubernetes.io/projected/96963007-afa9-4dfa-80a6-1ea9df272a6f-kube-api-access-qcn4t\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.104167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.104162    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96963007-afa9-4dfa-80a6-1ea9df272a6f-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.104167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.104172    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96963007-afa9-4dfa-80a6-1ea9df272a6f-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:43:37.139009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.138916    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:37.139009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.138978    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:37.288340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.288305    6430 generic.go:356] "Generic (PLEG): container finished" podID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerID="a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8" exitCode=0
Nov 03 23:43:37.288340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.288343    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"
Nov 03 23:43:37.288572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.288409    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" event={"ID":"96963007-afa9-4dfa-80a6-1ea9df272a6f","Type":"ContainerDied","Data":"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"}
Nov 03 23:43:37.288572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.288443    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52" event={"ID":"96963007-afa9-4dfa-80a6-1ea9df272a6f","Type":"ContainerDied","Data":"9c253b47caff7c46e81c25c53e1cafaf8e164908a0d2c03d3afe0a9d886313a9"}
Nov 03 23:43:37.288572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.288462    6430 scope.go:122] "RemoveContainer" containerID="a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"
Nov 03 23:43:37.298125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.298109    6430 scope.go:122] "RemoveContainer" containerID="a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"
Nov 03 23:43:37.298400 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:43:37.298373    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8\": container with ID starting with a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8 not found: ID does not exist" containerID="a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"
Nov 03 23:43:37.298457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.298414    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8"} err="failed to get container status \"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8\": rpc error: code = NotFound desc = could not find container \"a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8\": container with ID starting with a5179b862fd90aebf14f962d16896b0cd059250ba2f38883811cac4d092470b8 not found: ID does not exist"
Nov 03 23:43:37.342839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.342810    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:43:37.350237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:37.350211    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/e2e-conflict-a-86358670-kserve-6c8c5bfcf4-xfk52"]
Nov 03 23:43:38.138692 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:38.138656    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:38.138856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:38.138718    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:38.247339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:38.247295    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" path="/var/lib/kubelet/pods/96963007-afa9-4dfa-80a6-1ea9df272a6f/volumes"
Nov 03 23:43:39.139510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:39.139471    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:39.139696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:39.139533    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:40.138864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:40.138828    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:40.139264 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:40.138883    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:41.139127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:41.139091    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 23:43:41.139568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:41.139148    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 23:43:42.285842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:42.285377    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:42.286456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:42.286435    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:42.287876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:42.287700    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 03 23:43:44.427648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:44.427611    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:43:46.860438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:46.854503    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:43:46.918414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:46.917354    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.584529    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.627460    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.628307    6430 cpu_manager.go:414] "containerMap: removing container" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main"
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.628328    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main"
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.628543    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="96963007-afa9-4dfa-80a6-1ea9df272a6f" containerName="main"
Nov 03 23:43:47.743484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.672405    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:47.797628 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.797590    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 03 23:43:47.797945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:47.797924    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.058611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.058548    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:48.075674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.075610    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:43:48.076607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.076583    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:43:48.097021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.096993    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-80f8ba\"/\"distinct-2-w4-80f8ba-kserve-self-signed-certs\""
Nov 03 23:43:48.159950 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.078581    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:43:48.278413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.244739    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:43:48.359567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359446    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.359567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359538    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.359814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359604    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.359814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359708    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.359814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359760    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.359814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.359793    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7j7l\" (UniqueName: \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.427947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.427902    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:43:48.428259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.428241    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:43:48.461098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461061    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.461316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461302    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.461448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461434    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.461561 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461550    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.461642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461629    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.461710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.461699    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7j7l\" (UniqueName: \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.467102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.467068    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.476468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.470670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.476468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.471072    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.491867 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.491258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7j7l\" (UniqueName: \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.529706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.529670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.530240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.530213    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\") pod \"distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") " pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.550769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.550732    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:48.855949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:48.855923    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 03 23:43:48.858678 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:43:48.858649    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b371dfa_eaf6_4e45_a606_d641d9645dab.slice/crio-315961442fb17fc7b09e028ac248d22eeead146fdd13f5dfcc628bbed9340705 WatchSource:0}: Error finding container 315961442fb17fc7b09e028ac248d22eeead146fdd13f5dfcc628bbed9340705: Status 404 returned error can't find the container with id 315961442fb17fc7b09e028ac248d22eeead146fdd13f5dfcc628bbed9340705
Nov 03 23:43:49.670260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:49.670224    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" event={"ID":"0b371dfa-eaf6-4e45-a606-d641d9645dab","Type":"ContainerStarted","Data":"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"}
Nov 03 23:43:49.670723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:49.670276    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" event={"ID":"0b371dfa-eaf6-4e45-a606-d641d9645dab","Type":"ContainerStarted","Data":"315961442fb17fc7b09e028ac248d22eeead146fdd13f5dfcc628bbed9340705"}
Nov 03 23:43:49.693843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:49.693785    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podStartSLOduration=2.693771304 podStartE2EDuration="2.693771304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:49.692175006 +0000 UTC m=+845.583236941" watchObservedRunningTime="2026-11-03 23:43:49.693771304 +0000 UTC m=+845.584833285"
Nov 03 23:43:50.555223 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:50.555180    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:58.555252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:58.555211    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:58.568460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:58.568427    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:43:58.746279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:43:58.746250    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:44:02.540421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.539994    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 03 23:44:02.553372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.553335    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558503    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558559    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558591    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558617    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558656    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558726    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558895    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558923    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.569415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.558975    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h277\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.583689 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.583648    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777313    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777475    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777508    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777552    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h277\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777640    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777669    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777695    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.777732    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.779051    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.779422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.779318    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.780153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.780009    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.783419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.780671    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.783419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.780903    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.807409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.806378    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.940567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.940497    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.961006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.940875    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:02.961006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:02.956129    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h277\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:03.218463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.214471    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:03.806353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.806289    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 03 23:44:03.812845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.812803    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:03.812951 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.812898    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:03.812951 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.812944    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:03.955106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.955063    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" event={"ID":"562e9478-dd97-45bb-8e99-c01fd16b9b5b","Type":"ContainerStarted","Data":"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"}
Nov 03 23:44:03.955269 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.955133    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" event={"ID":"562e9478-dd97-45bb-8e99-c01fd16b9b5b","Type":"ContainerStarted","Data":"c22df7ec35d32448124900a64efc51473057b8b72d2ae55608436159a85c902a"}
Nov 03 23:44:03.985320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:03.985265    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podStartSLOduration=1.9852518080000001 podStartE2EDuration="1.985251808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:03.982889226 +0000 UTC m=+859.873951184" watchObservedRunningTime="2026-11-03 23:44:03.985251808 +0000 UTC m=+859.876313850"
Nov 03 23:44:04.215485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:04.215384    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:04.217096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:04.217068    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:04.217266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:04.217136    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:04.894357 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:44:04.894320    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:44:05.222503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:05.215773    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:05.222503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:05.215835    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:06.233903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:06.233846    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:06.234426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:06.233925    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:07.233526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.233488    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:07.233720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.233547    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:07.839534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.839499    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:44:07.844685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.844660    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.861424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.861380    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:44:07.898296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898261    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898322    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898404    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898433    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898452    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898493    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898701    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898738    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:07.898959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:07.898931    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtbv\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000254    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtbv\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000338    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000372    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000418    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000441    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000459    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000503    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000609    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000646    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.000993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.000971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.001099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.001071    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.001185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.001168    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.001247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.001209    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.001358 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.001340    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.003425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.003399    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.003617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.003600    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.033252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.033150    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.033857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.033834    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtbv\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.163201 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.163103    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:08.229415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.220725    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:08.229415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.220785    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:08.544181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.543858    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:44:08.548460 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:44:08.547550    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724b281b_1a95_47fd_a1c8_acec7a88d2f7.slice/crio-a1c4f84914afd7bcfa255d2f247038c9ddfab8b7006eeb86db0fb2d1e82fd710 WatchSource:0}: Error finding container a1c4f84914afd7bcfa255d2f247038c9ddfab8b7006eeb86db0fb2d1e82fd710: Status 404 returned error can't find the container with id a1c4f84914afd7bcfa255d2f247038c9ddfab8b7006eeb86db0fb2d1e82fd710
Nov 03 23:44:08.551215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.551165    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:08.551300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.551250    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:08.551300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.551283    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:08.983922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.983876    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" event={"ID":"724b281b-1a95-47fd-a1c8-acec7a88d2f7","Type":"ContainerStarted","Data":"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"}
Nov 03 23:44:08.984460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:08.983938    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" event={"ID":"724b281b-1a95-47fd-a1c8-acec7a88d2f7","Type":"ContainerStarted","Data":"a1c4f84914afd7bcfa255d2f247038c9ddfab8b7006eeb86db0fb2d1e82fd710"}
Nov 03 23:44:09.163684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:09.163565    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:09.165341 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:09.165307    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:09.165519 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:09.165406    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:09.223412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:09.218840    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:09.223412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:09.218913    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:10.164371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:10.164326    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:10.164930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:10.164414    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:10.218424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:10.217716    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 23:44:10.218424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:10.217777    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 23:44:11.164461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.164419    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:11.164957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.164480    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:11.237751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.237717    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:11.238428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.238357    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:11.239250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.239230    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 03 23:44:11.272008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:11.271958    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podStartSLOduration=4.271943116 podStartE2EDuration="4.271943116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:09.021735402 +0000 UTC m=+864.912797337" watchObservedRunningTime="2026-11-03 23:44:11.271943116 +0000 UTC m=+867.163005040"
Nov 03 23:44:12.164419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:12.164355    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:12.164615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:12.164440    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:13.164308 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.164272    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:13.164492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.164326    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:13.340441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.340398    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 03 23:44:13.344332 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.344307    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.357595 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.357570    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"openshift-service-ca.crt\""
Nov 03 23:44:13.357931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.357903    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"kube-root-ca.crt\""
Nov 03 23:44:13.358379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.358250    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"default-dockercfg-926zp\""
Nov 03 23:44:13.358570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.358274    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"distinct-w0-f6f292-kserve-self-signed-certs\""
Nov 03 23:44:13.359539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359511    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.359651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.359789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359763    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zdr\" (UniqueName: \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.359850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359792    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.359902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359887    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.359963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.359918    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.360605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.360581    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 03 23:44:13.460958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.460866    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.460958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.460937    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461007    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zdr\" (UniqueName: \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461032    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461068    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461108    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461323    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461375    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.461534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.461513    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.463366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.463342    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.463596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.463576    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.474211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.474186    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zdr\" (UniqueName: \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\") pod \"distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:13.666981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:13.666944    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:14.004252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:14.004223    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 03 23:44:14.005725 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:44:14.005687    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc6f5e45_84ad_46f4_821d_75e9916977d5.slice/crio-d0628c7904e8ea4102bf4d480715c8e1b03024e0a6c3bc2a1d0aa21c395bd74e WatchSource:0}: Error finding container d0628c7904e8ea4102bf4d480715c8e1b03024e0a6c3bc2a1d0aa21c395bd74e: Status 404 returned error can't find the container with id d0628c7904e8ea4102bf4d480715c8e1b03024e0a6c3bc2a1d0aa21c395bd74e
Nov 03 23:44:14.039596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:14.038811    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" event={"ID":"bc6f5e45-84ad-46f4-821d-75e9916977d5","Type":"ContainerStarted","Data":"d0628c7904e8ea4102bf4d480715c8e1b03024e0a6c3bc2a1d0aa21c395bd74e"}
Nov 03 23:44:14.163703 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:14.163670    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:14.163879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:14.163725    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:15.046819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:15.046776    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" event={"ID":"bc6f5e45-84ad-46f4-821d-75e9916977d5","Type":"ContainerStarted","Data":"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"}
Nov 03 23:44:15.084470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:15.084369    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podStartSLOduration=2.084352214 podStartE2EDuration="2.084352214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:15.083472695 +0000 UTC m=+870.974534629" watchObservedRunningTime="2026-11-03 23:44:15.084352214 +0000 UTC m=+870.975414148"
Nov 03 23:44:15.164468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:15.164427    6430 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 23:44:15.164655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:15.164488    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 23:44:15.671411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:15.670962    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:16.169672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:16.169641    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:16.170205 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:16.170157    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:16.171159 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:16.171138    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:44:20.383873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:20.383823    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:44:23.674449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:23.669378    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:23.706668 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:23.706364    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:24.133368 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:24.133333    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:27.787180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.787145    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 03 23:44:27.793562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.793537    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.802010 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.801988    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"distinct-2-w0-f6f292-kserve-self-signed-certs\""
Nov 03 23:44:27.808768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.808742    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 03 23:44:27.845009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.844972    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.845170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.845102    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.845272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.845252    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd69j\" (UniqueName: \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.845353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.845340    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.845455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.845434    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.845497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.845471    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946490    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd69j\" (UniqueName: \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946571    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946607    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946640    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946696    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.946922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946777    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.947011 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.946992    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.947073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.947034    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.947174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.947154    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.948918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.948897    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.949194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.949176    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:27.996417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:27.989140    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd69j\" (UniqueName: \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\") pod \"distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") " pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:28.112511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:28.112471    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:28.438463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:28.428168    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 03 23:44:29.156321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:29.156279    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" event={"ID":"6b105d04-a066-4e09-92f5-99511832970a","Type":"ContainerStarted","Data":"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"}
Nov 03 23:44:29.156746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:29.156332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" event={"ID":"6b105d04-a066-4e09-92f5-99511832970a","Type":"ContainerStarted","Data":"fae509db496ed7f0d23338f1331c7d4d9ec1284cb47db2d6e20993b461f34fc5"}
Nov 03 23:44:29.181889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:29.181839    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podStartSLOduration=2.181822428 podStartE2EDuration="2.181822428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:29.180103801 +0000 UTC m=+885.071165738" watchObservedRunningTime="2026-11-03 23:44:29.181822428 +0000 UTC m=+885.072884395"
Nov 03 23:44:30.113943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:30.113903    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:34.831353 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:44:34.831308    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:44:38.113474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:38.113440    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:38.126660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:38.126629    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:38.219036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:38.219009    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:41.347413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:41.346465    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:41.627513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:41.625866    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:44:41.714413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:41.712248    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:41.746413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:41.745780    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:41.864415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:41.863487    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.137495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.136532    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.236419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.233498    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.522474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.426930    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.522474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.427319    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:44:42.628984 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.627780    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.742590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.742553    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:44:42.742883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.742866    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:44:42.791593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.791480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.796230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.791833    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.821599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.811344    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:44:42.821599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.811430    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:44:42.821599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.791862    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:44:42.889590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.866487    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:42.951981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:42.889920    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:44:43.097037 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.096985    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.203814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.138317    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.233508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.233472    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:43.233796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.233757    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.234467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.138798    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:43.234467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.234288    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.236228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.236206    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:43.397244 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.386102    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:44:43.397244 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.386169    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.523568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.523521    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.537113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.537069    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.537795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.537769    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:44:43.677497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.669903    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.810541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.810497    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.835734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.810788    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:44:43.836452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.813897    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.850515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.824668    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:44:43.850515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.840772    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:44:43.883473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.852998    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:44:43.883473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.868673    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:44:43.883473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.869546    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:44:43.894954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.894926    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:44:43.906407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.896147    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Nov 03 23:44:43.915047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.914953    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:44:43.922005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.921981    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:44:43.948200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.948100    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:44:43.968195 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:43.968165    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:44:44.002559 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:44.002530    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:44:47.589825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.589782    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:44:47.597759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.597732    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.603073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.603030    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:44:47.624924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.624889    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:44:47.628879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.628860    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.646256 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.646229    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:44:47.658109 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658080    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658235 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658121    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658235 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkzh\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658268    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wjx\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658459    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658507    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658530    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658550    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658755 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658590    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658755 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658610    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.658854 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658771    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658872    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.658976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658959    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.659039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.658990    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.659039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.659026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.659145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.659067    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760170    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760270    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760332    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760357    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760432    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760509    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760542    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760564    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkzh\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760625    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wjx\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760678    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760735    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760785    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760797    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760864    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760874    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.760921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760898    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.761314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.761314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.760966    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.761314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761174    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.761314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761259    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.761546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.761669 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761644    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.761917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761889    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.762016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761954    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.762016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.761954    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.762264 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.762236    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.763984 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.763949    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.764089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.764056    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.764140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.764092    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.764250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.764229    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.777582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.777551    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkzh\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.778450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.778425    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.779054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.779031    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.779171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.779028    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wjx\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:47.915878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.915782    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:47.947723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:47.947692    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:48.234522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.233569    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:44:48.234986 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:44:48.234956    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702cdb11_4642_4c8c_a286_adc7aa174473.slice/crio-86396d3c87e4720b0158e4efccf1acb84bc185ee508fcdae296feadc837a5c5a WatchSource:0}: Error finding container 86396d3c87e4720b0158e4efccf1acb84bc185ee508fcdae296feadc837a5c5a: Status 404 returned error can't find the container with id 86396d3c87e4720b0158e4efccf1acb84bc185ee508fcdae296feadc837a5c5a
Nov 03 23:44:48.237329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.237295    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.237490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.237375    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.237555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.237531    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.257103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.257076    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:44:48.258627 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:44:48.258599    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27977e73_bc5d_4951_bf7a_35ef9f157302.slice/crio-a5b4d6c7fd646ac910c131a435f43a4128b26fa5ebcd9d23e1a82fb022900045 WatchSource:0}: Error finding container a5b4d6c7fd646ac910c131a435f43a4128b26fa5ebcd9d23e1a82fb022900045: Status 404 returned error can't find the container with id a5b4d6c7fd646ac910c131a435f43a4128b26fa5ebcd9d23e1a82fb022900045
Nov 03 23:44:48.260914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.260884    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.260991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.260945    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.260991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.260977    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:44:48.789147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.789115    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" event={"ID":"27977e73-bc5d-4951-bf7a-35ef9f157302","Type":"ContainerStarted","Data":"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"}
Nov 03 23:44:48.789563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.789173    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" event={"ID":"27977e73-bc5d-4951-bf7a-35ef9f157302","Type":"ContainerStarted","Data":"a5b4d6c7fd646ac910c131a435f43a4128b26fa5ebcd9d23e1a82fb022900045"}
Nov 03 23:44:48.790684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.790660    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" event={"ID":"702cdb11-4642-4c8c-a286-adc7aa174473","Type":"ContainerStarted","Data":"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"}
Nov 03 23:44:48.790798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.790699    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" event={"ID":"702cdb11-4642-4c8c-a286-adc7aa174473","Type":"ContainerStarted","Data":"86396d3c87e4720b0158e4efccf1acb84bc185ee508fcdae296feadc837a5c5a"}
Nov 03 23:44:48.822417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.822067    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podStartSLOduration=1.822048719 podStartE2EDuration="1.822048719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:48.820927543 +0000 UTC m=+904.711989479" watchObservedRunningTime="2026-11-03 23:44:48.822048719 +0000 UTC m=+904.713110656"
Nov 03 23:44:48.853409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.852414    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podStartSLOduration=1.852377216 podStartE2EDuration="1.852377216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:48.849922826 +0000 UTC m=+904.740984785" watchObservedRunningTime="2026-11-03 23:44:48.852377216 +0000 UTC m=+904.743439153"
Nov 03 23:44:48.924414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.918545    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:48.955416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:48.953483    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:49.932414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:49.929125    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:49.932414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:49.929185    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:49.961549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:49.960633    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:49.961549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:49.960693    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:50.939541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:50.939495    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:50.940130 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:50.940092    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:50.949307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:50.949275    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:50.949591 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:50.949562    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:51.930109 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:51.930071    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:51.930431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:51.930380    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:51.952020 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:51.951982    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:51.952666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:51.952630    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:52.931411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:52.928228    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:52.931411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:52.928289    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:52.962761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:52.962725    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:52.963460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:52.963426    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:53.945471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:53.944864    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:53.945471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:53.944938    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:53.957413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:53.949771    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:53.957413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:53.949839    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:54.916695 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:54.916653    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 23:44:54.917329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:54.917294    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 23:44:54.953854 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:54.953810    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 23:44:54.954140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:54.954107    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 23:44:55.970409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.967684    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:55.970409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.969080    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:55.970409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.969476    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:55.970409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.970247    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:55.979420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.977813    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:44:55.979420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:55.978136    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:44:58.322658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:44:58.322624    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:45:00.170410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.167355    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:00.176743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.176714    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:00.177051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.177030    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.197573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.197540    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 03 23:45:00.295053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.294986    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxmn\" (UniqueName: \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.295458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.295430    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.406420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.400203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxmn\" (UniqueName: \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.406420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.400351    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.406420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.401412    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.591439 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.574702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxmn\" (UniqueName: \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.601941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.601618    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:00.699489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.699227    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 03 23:45:00.802414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.774816    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:00.826581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.826523    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:00.909629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.827950    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:00.978978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.978939    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.052605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.052564    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.077412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.052886    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.158659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.158613    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.189817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.189717    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.190464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.190432    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.190706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.190681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qnt\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.190869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.190851    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.191286 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:00.832709    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 03 23:45:01.300295 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.300241    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.300646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.300623    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.300801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.300788    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.300958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.300945    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301084    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301217    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qnt\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301345 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301332    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301610    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.301866 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.301838    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.302170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.302149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.302756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.302449    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.315536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.315503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.317713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.317687    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.346191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.346159    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.354724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.354656    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.356058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.356027    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.371467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.356865    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qnt\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.543799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.543709    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:01.831472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.767623    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.025419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.903897    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:45:02.025419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.904513    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:45:02.025419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.956485    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.025419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.957497    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:45:02.025419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:01.957783    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.025851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.025480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.687776 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.687310    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.830713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.830659    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:02.926415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.872427    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.995189    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:02.999942    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.000972    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.014174    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.022621    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.057012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.036744    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.115734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.071588    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.581163 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:03.581126    6430 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.322s"
Nov 03 23:45:03.581525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.581505    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:45:03.581699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.581685    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:03.581849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.581836    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:45:03.581982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.581970    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:45:03.582105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.582094    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:45:03.582278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.582263    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:45:03.697009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.696966    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:03.697573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.697542    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:03.697735 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.697715    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.863757    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.886485    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.886957    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.912692    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.913019    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:45:03.923466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.913427    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:45:03.966573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:03.966458    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 03 23:45:04.054425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.051026    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.054425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.051472    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.054425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.051746    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.087600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.087551    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:45:04.088199 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.088166    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:04.088669 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.088650    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:45:04.190460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.188626    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:45:04.407422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.405871    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:45:04.410790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.410746    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.697417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.697358    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.735288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.735256    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:45:04.761154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.761117    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerStarted","Data":"5f5ebe77840ef8199293b40cdee01b5fcee7381e2aff0ea9091bc87fb39cb2fc"}
Nov 03 23:45:04.900494 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.900444    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"]
Nov 03 23:45:04.926412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.919543    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:04.994471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:04.973235    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:45:05.106797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.106749    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.231541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.203043    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" event={"ID":"8abf9512-7c01-4031-a73f-f6fae8939179","Type":"ContainerStarted","Data":"963830819f023bdbef6bdd108fb6f2791ae788231a5e29f820272ad69d7ba526"}
Nov 03 23:45:05.231541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.203103    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"]
Nov 03 23:45:05.231541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.203210    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.231859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.231586    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:45:05.231988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.231956    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:05.253511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.242687    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 23:45:05.253511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.247991    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:45:05.418418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.382624    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.455821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.455778    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvk2\" (UniqueName: \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.468655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.467238    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.469180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.468650    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.499184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.499153    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:45:05.603717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.603675    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.603944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.603919    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvk2\" (UniqueName: \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.604185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.604158    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.639714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.616532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.661545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.661498    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:05.679839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.679764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.714636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.714195    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvk2\" (UniqueName: \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\") pod \"maas-api-revoke-sub-8fcced1e-9dkgh\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.764763 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:05.760036    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:45:05.782676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.782643    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:05.833334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.832235    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:45:05.855741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.854502    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:45:05.900426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.899956    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:45:05.937443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.937084    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:45:05.939544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.938958    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:45:05.979691 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:05.978752    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" event={"ID":"8abf9512-7c01-4031-a73f-f6fae8939179","Type":"ContainerStarted","Data":"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"}
Nov 03 23:45:06.143992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.143929    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podStartSLOduration=6.143910294 podStartE2EDuration="6.143910294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:06.14206378 +0000 UTC m=+922.033125715" watchObservedRunningTime="2026-11-03 23:45:06.143910294 +0000 UTC m=+922.034972229"
Nov 03 23:45:06.341504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.333985    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"]
Nov 03 23:45:06.346668 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:06.346633    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c8f878_422a_44aa_bc86_4361e652f713.slice/crio-705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22 WatchSource:0}: Error finding container 705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22: Status 404 returned error can't find the container with id 705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22
Nov 03 23:45:06.545360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.545273    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:06.547007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.546976    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:06.547125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.547047    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:06.986752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:06.986711    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh" event={"ID":"22c8f878-422a-44aa-bc86-4361e652f713","Type":"ContainerStarted","Data":"705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22"}
Nov 03 23:45:07.545039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:07.545010    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:07.545205 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:07.545062    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:08.545457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:08.545421    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:08.545944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:08.545478    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:09.545265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:09.545231    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:09.545437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:09.545294    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:10.034769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.034737    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerStarted","Data":"f74522b3bded7b32cd2114577f4c4c7a27038e2630e00718644bf2c149227c4d"}
Nov 03 23:45:10.037748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.037713    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh" event={"ID":"22c8f878-422a-44aa-bc86-4361e652f713","Type":"ContainerStarted","Data":"4618b0e230d9b72721cd8b3a5e33183b5823e999cd88f01e17dcb0d05d127852"}
Nov 03 23:45:10.056864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.056818    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" podStartSLOduration=3.945200811 podStartE2EDuration="10.056804365s" totalImagesPullingTime="6.111603554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:10.055612754 +0000 UTC m=+925.946674691" watchObservedRunningTime="2026-11-03 23:45:10.056804365 +0000 UTC m=+925.947866299"
Nov 03 23:45:10.084238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.084193    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh" podStartSLOduration=2.94988499 podStartE2EDuration="6.084179603s" totalImagesPullingTime="3.134294613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:10.080558046 +0000 UTC m=+925.971619985" watchObservedRunningTime="2026-11-03 23:45:10.084179603 +0000 UTC m=+925.975241559"
Nov 03 23:45:10.545045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.545011    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:10.545243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:10.545071    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:11.046209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.046171    6430 generic.go:356] "Generic (PLEG): container finished" podID="12b20923-8a7e-4489-bb37-a7df923b283c" containerID="f74522b3bded7b32cd2114577f4c4c7a27038e2630e00718644bf2c149227c4d" exitCode=60
Nov 03 23:45:11.046663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.046257    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerDied","Data":"f74522b3bded7b32cd2114577f4c4c7a27038e2630e00718644bf2c149227c4d"}
Nov 03 23:45:11.046663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.046568    6430 scope.go:122] "RemoveContainer" containerID="f74522b3bded7b32cd2114577f4c4c7a27038e2630e00718644bf2c149227c4d"
Nov 03 23:45:11.048284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.048237    6430 generic.go:356] "Generic (PLEG): container finished" podID="22c8f878-422a-44aa-bc86-4361e652f713" containerID="4618b0e230d9b72721cd8b3a5e33183b5823e999cd88f01e17dcb0d05d127852" exitCode=0
Nov 03 23:45:11.048284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.048275    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh" event={"ID":"22c8f878-422a-44aa-bc86-4361e652f713","Type":"ContainerDied","Data":"4618b0e230d9b72721cd8b3a5e33183b5823e999cd88f01e17dcb0d05d127852"}
Nov 03 23:45:11.545355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.545318    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:11.545552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:11.545377    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:12.058448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.058378    6430 generic.go:356] "Generic (PLEG): container finished" podID="12b20923-8a7e-4489-bb37-a7df923b283c" containerID="38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad" exitCode=60
Nov 03 23:45:12.058890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.058510    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerDied","Data":"38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad"}
Nov 03 23:45:12.058890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.058571    6430 scope.go:122] "RemoveContainer" containerID="f74522b3bded7b32cd2114577f4c4c7a27038e2630e00718644bf2c149227c4d"
Nov 03 23:45:12.058890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.058842    6430 scope.go:122] "RemoveContainer" containerID="38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad"
Nov 03 23:45:12.059217 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:12.059192    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k_odh-ai-gateway-infra(12b20923-8a7e-4489-bb37-a7df923b283c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" podUID="12b20923-8a7e-4489-bb37-a7df923b283c"
Nov 03 23:45:12.241721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.241685    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:12.398687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.398648    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\" (UniqueName: \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\") pod \"22c8f878-422a-44aa-bc86-4361e652f713\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") "
Nov 03 23:45:12.398897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.398736    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\") pod \"22c8f878-422a-44aa-bc86-4361e652f713\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") "
Nov 03 23:45:12.398897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.398841    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\") pod \"22c8f878-422a-44aa-bc86-4361e652f713\" (UID: \"22c8f878-422a-44aa-bc86-4361e652f713\") "
Nov 03 23:45:12.399673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.399546    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca" pod "22c8f878-422a-44aa-bc86-4361e652f713" (UID: "22c8f878-422a-44aa-bc86-4361e652f713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:12.401335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.400694    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.412468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.412417    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2" pod "22c8f878-422a-44aa-bc86-4361e652f713" (UID: "22c8f878-422a-44aa-bc86-4361e652f713"). InnerVolumeSpecName "kube-api-access-rnvk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:12.414059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.414023    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca" pod "22c8f878-422a-44aa-bc86-4361e652f713" (UID: "22c8f878-422a-44aa-bc86-4361e652f713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:45:12.502032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.501989    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnvk2\" (UniqueName: \"kubernetes.io/projected/22c8f878-422a-44aa-bc86-4361e652f713-kube-api-access-rnvk2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.502032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.502031    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8f878-422a-44aa-bc86-4361e652f713-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:12.545368 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.545335    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 23:45:12.545566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:12.545419    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 23:45:13.067741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.067702    6430 scope.go:122] "RemoveContainer" containerID="38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad"
Nov 03 23:45:13.068279 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:13.068072    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k_odh-ai-gateway-infra(12b20923-8a7e-4489-bb37-a7df923b283c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" podUID="12b20923-8a7e-4489-bb37-a7df923b283c"
Nov 03 23:45:13.069323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.069293    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh" event={"ID":"22c8f878-422a-44aa-bc86-4361e652f713","Type":"ContainerDied","Data":"705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22"}
Nov 03 23:45:13.069484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.069377    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705c321283de8d054c0be87282cf5a357ac03ca0610c5ed417e2e64653e9df22"
Nov 03 23:45:13.069484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.069463    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"
Nov 03 23:45:13.548703 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.548675    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:13.549079 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.549055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:13.549818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:13.549799    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 03 23:45:14.299145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.299107    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 03 23:45:14.299980 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.299956    6430 cpu_manager.go:414] "containerMap: removing container" podUID="22c8f878-422a-44aa-bc86-4361e652f713" containerName="revoke-keys"
Nov 03 23:45:14.300099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.299982    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c8f878-422a-44aa-bc86-4361e652f713" containerName="revoke-keys"
Nov 03 23:45:14.300163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.300143    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c8f878-422a-44aa-bc86-4361e652f713" containerName="revoke-keys"
Nov 03 23:45:14.303751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.303729    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.316926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.316898    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 03 23:45:14.439941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.439904    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.439979    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440096    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440218    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440346 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440332    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440404 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440374    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440401    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440421    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jjdl\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.440574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.440554    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.548452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.548349    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.549407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.548886    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.549663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.549638    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.549838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.549804    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.549967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.549951    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jjdl\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.550114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.550088    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.550440 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.550293    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.550616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.550597    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.550785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.550759    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.552033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.552005    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.552124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.552037    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.552124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.552087    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.552292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.552263    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.552752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.552730    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.554074 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.554046    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.554580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.554561    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.567338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.567305    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.567519 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.567503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jjdl\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.619501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.619470    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:14.913511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.913481    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 03 23:45:14.913622 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:14.913598    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24df8d1_00b8_4c1f_a891_d5110cec430f.slice/crio-cc1afdfd4ac6eaeaaacb22356f6c31397d8d1b9036b43c262441b81615786777 WatchSource:0}: Error finding container cc1afdfd4ac6eaeaaacb22356f6c31397d8d1b9036b43c262441b81615786777: Status 404 returned error can't find the container with id cc1afdfd4ac6eaeaaacb22356f6c31397d8d1b9036b43c262441b81615786777
Nov 03 23:45:14.915989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.915961    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:14.916051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.916021    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:14.916084 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:14.916049    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:15.087220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.087183    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" event={"ID":"e24df8d1-00b8-4c1f-a891-d5110cec430f","Type":"ContainerStarted","Data":"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"}
Nov 03 23:45:15.087410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.087240    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" event={"ID":"e24df8d1-00b8-4c1f-a891-d5110cec430f","Type":"ContainerStarted","Data":"cc1afdfd4ac6eaeaaacb22356f6c31397d8d1b9036b43c262441b81615786777"}
Nov 03 23:45:15.116051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.116002    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podStartSLOduration=1.115989293 podStartE2EDuration="1.115989293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:15.111776117 +0000 UTC m=+931.002838053" watchObservedRunningTime="2026-11-03 23:45:15.115989293 +0000 UTC m=+931.007051227"
Nov 03 23:45:15.626412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.622119    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:15.632548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.632512    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:15.632719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:15.632569    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:16.622412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:16.621732    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:16.622412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:16.621794    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:17.621840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:17.621798    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:17.632413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:17.622794    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:18.630416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:18.627834    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:18.630416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:18.627901    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:19.626414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:19.621816    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:19.626414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:19.621898    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:20.632275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:20.631851    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:20.632275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:20.631935    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:21.620311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:21.620274    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 23:45:21.620501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:21.620328    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 23:45:22.258829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.258794    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:45:22.269070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.269024    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:45:22.269852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.269750    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.310776 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.310728    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88f34dbe-dockercfg-h9zmb\""
Nov 03 23:45:22.321319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.321297    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88f34dbe\""
Nov 03 23:45:22.388322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.388288    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.388322 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.388324    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.498649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.490307    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.498649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.490358    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.498649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.491799    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.528461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.528096    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.610087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.609002    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:22.648442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.642015    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:22.648442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.643606    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:22.677407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.662023    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 03 23:45:22.963061 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.963034    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:45:22.964434 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:22.964411    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999d7568_59ab_4ef6_b3d1_98dc2d81c5f2.slice/crio-fd8029f18e7b0a03580a6bc283175a57d9a24cb264a4777756cb94e72d350158 WatchSource:0}: Error finding container fd8029f18e7b0a03580a6bc283175a57d9a24cb264a4777756cb94e72d350158: Status 404 returned error can't find the container with id fd8029f18e7b0a03580a6bc283175a57d9a24cb264a4777756cb94e72d350158
Nov 03 23:45:22.967108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:22.967089    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:45:23.170765 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:23.170726    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" event={"ID":"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2","Type":"ContainerStarted","Data":"fd8029f18e7b0a03580a6bc283175a57d9a24cb264a4777756cb94e72d350158"}
Nov 03 23:45:23.224847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:23.224820    6430 scope.go:122] "RemoveContainer" containerID="38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad"
Nov 03 23:45:24.181917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:24.181884    6430 generic.go:356] "Generic (PLEG): container finished" podID="12b20923-8a7e-4489-bb37-a7df923b283c" containerID="fdbd8c2fc51e09bf02c2c4339c129068aa4677ad80da06d442036f5bd51d1e12" exitCode=60
Nov 03 23:45:24.182323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:24.181972    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerDied","Data":"fdbd8c2fc51e09bf02c2c4339c129068aa4677ad80da06d442036f5bd51d1e12"}
Nov 03 23:45:24.182323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:24.182040    6430 scope.go:122] "RemoveContainer" containerID="38604a14ee79ad764249b29695b24849528f5432633a9c875700062338958fad"
Nov 03 23:45:24.182649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:24.182630    6430 scope.go:122] "RemoveContainer" containerID="fdbd8c2fc51e09bf02c2c4339c129068aa4677ad80da06d442036f5bd51d1e12"
Nov 03 23:45:24.182958 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:24.182933    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k_odh-ai-gateway-infra(12b20923-8a7e-4489-bb37-a7df923b283c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" podUID="12b20923-8a7e-4489-bb37-a7df923b283c"
Nov 03 23:45:25.264411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:25.256178    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:26.528921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:26.510156    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:26.744115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:26.744031    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:26.951621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:26.951562    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:26.990786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:26.960859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\" (UniqueName: \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\") pod \"12b20923-8a7e-4489-bb37-a7df923b283c\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") "
Nov 03 23:45:26.991483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:26.991445    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\") pod \"12b20923-8a7e-4489-bb37-a7df923b283c\" (UID: \"12b20923-8a7e-4489-bb37-a7df923b283c\") "
Nov 03 23:45:27.014468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.001621    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca" pod "12b20923-8a7e-4489-bb37-a7df923b283c" (UID: "12b20923-8a7e-4489-bb37-a7df923b283c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:45:27.080452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.059817    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:45:27.107594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.107549    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b20923-8a7e-4489-bb37-a7df923b283c-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:27.135912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.135863    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn" pod "12b20923-8a7e-4489-bb37-a7df923b283c" (UID: "12b20923-8a7e-4489-bb37-a7df923b283c"). InnerVolumeSpecName "kube-api-access-hlxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:45:27.212624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.212580    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxmn\" (UniqueName: \"kubernetes.io/projected/12b20923-8a7e-4489-bb37-a7df923b283c-kube-api-access-hlxmn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:45:27.223497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.219556    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:45:27.292406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.292292    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:27.308490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.307779    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:27.308490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.307934    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.527533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.527479    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:45:27.528919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.528885    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.529369 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.529350    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:45:27.629601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.629556    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:27.772494 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.772447    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:45:27.773739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.773708    6430 cpu_manager.go:414] "containerMap: removing container" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.782479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.782438    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.807492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.807451    6430 cpu_manager.go:414] "containerMap: removing container" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.807721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.807697    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.843222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.807844    6430 cpu_manager.go:414] "containerMap: removing container" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.871586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.860060    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.871586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.860357    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.871586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.860382    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:27.871586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.860486    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" containerName="cleanup"
Nov 03 23:45:28.007725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.987736    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:45:28.007725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.987874    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.007725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:27.999472    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.007725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.000085    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:45:28.088558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.088516    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-344261-openshift-default-dockercfg-gpvqj\""
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.095593    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.096025    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.096633    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.097785    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.100006    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k" event={"ID":"12b20923-8a7e-4489-bb37-a7df923b283c","Type":"ContainerDied","Data":"5f5ebe77840ef8199293b40cdee01b5fcee7381e2aff0ea9091bc87fb39cb2fc"}
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.100067    6430 scope.go:122] "RemoveContainer" containerID="fdbd8c2fc51e09bf02c2c4339c129068aa4677ad80da06d442036f5bd51d1e12"
Nov 03 23:45:28.115431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.100208    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132570    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132704    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132732    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132789    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132842    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdsr\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132889    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.132966    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.133086    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.144432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.133119    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.148290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.146710    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.148290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.146794    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240376    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240521    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240560    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240705    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240818    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240869    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240933    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdsr\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.240983    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.241915    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.242240    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.261512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.242908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.320539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.311754    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.349414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.321162    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:28.418572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.386575    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:28.418572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.328582    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.419749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.419690    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.420338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.420317    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:28.584510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.584470    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.623421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.612637    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:45:28.646830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.646802    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:45:28.758720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.727327    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:45:28.785725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.762162    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:45:28.858476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:28.858328    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:45:29.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.009716    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:29.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.011003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:29.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.020125    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdsr\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:29.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.020681    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.173592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.173489    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.198706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.173995    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:45:29.198706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.174951    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:45:29.198706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.175020    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:45:29.198706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.175995    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.199946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.199215    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.560413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.559913    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:29.757509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.756940    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:29.771536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.771495    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\") pod \"e2e-api-a-344261-openshift-default-85f479895c-tncch\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") " pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:29.911519 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.882553    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:45:30.067769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.911928    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.068763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:29.914108    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.079149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.079109    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.136929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.136880    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.248584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.248521    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.266490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.266326    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:45:30.287483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.270164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" podUID="bfcab315-9f9c-4214-9fd7-36811b39c65b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.382055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.381947    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:45:30.475063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.451026    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:30.525814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.525763    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:45:30.621984 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.621939    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:30.745522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:30.734819    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 03 23:45:31.259415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.243295    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:45:31.259910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.259426    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:31.267179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.267141    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:31.356414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.351753    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:45:31.356414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.352106    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:31.356414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.354313    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:45:31.409209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.409169    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-n9b2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:45:31.409492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.409455    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:31.959966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:31.959904    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:32.039261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.039166    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:45:32.259609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.259322    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:32.270424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.270039    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:32.281593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.281541    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:32.356540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.356495    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:45:32.357574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.357542    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:45:32.358357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.358334    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:45:32.467251 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:32.465372    6430 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.241s"
Nov 03 23:45:32.467251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.466017    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:45:32.467251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.466040    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:32.638612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.638578    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895825-2xd9k"]
Nov 03 23:45:32.638915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.638894    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:45:32.677424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.677369    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:45:32.868336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.868196    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:45:32.873208 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:32.873178    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:45:33.096453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.093654    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.211459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.181911    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.247416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.212588    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-953960-openshift-default-dockercfg-rb5jt\""
Nov 03 23:45:33.247416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.223331    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:45:33.331330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.296687    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.357412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.357352    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.357923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.357885    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.358609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.358575    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.435432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.435313    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:33.438094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.437893    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:33.439506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.439469    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:33.452736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.452601    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.453166 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.453139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.453398 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.453362    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.453548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.453523    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.453805 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.453786    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.454091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.454074    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.454242 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.454227    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.454445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.454365    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rmb\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.454616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.454572    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.500632 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.438355    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.556684    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.556752    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.556803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.556954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.557021    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.557074    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rmb\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.557114    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.557162    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.557338    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.558069    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.651921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.558298    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.668451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.558371    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.668943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.571911    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.669138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.572212    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.669272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.594419    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.669504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.598256    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:45:33.671288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.663975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.764515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.695862    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerStarted","Data":"7b54370fc078574fc2b83e9c428d3bf8cfdf9abae1aaf251be4df7852b9c7d9e"}
Nov 03 23:45:33.831213 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.831183    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:45:33.863070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.803677    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rmb\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:33.998229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:33.998149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\") pod \"e2e-api-a-953960-openshift-default-7545d8768f-qbqbf\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") " pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:34.007536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.006956    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:45:34.010184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.010068    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:45:34.019402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.012900    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:45:34.024788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.024768    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:45:34.239521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.239483    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b20923-8a7e-4489-bb37-a7df923b283c" path="/var/lib/kubelet/pods/12b20923-8a7e-4489-bb37-a7df923b283c/volumes"
Nov 03 23:45:34.243430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.240642    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:34.565979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.565949    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:45:34.567423 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:34.567346    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9526b9_75de_4674_9b33_1af6ee23abfa.slice/crio-2a6513f41c418bdbfe9e1d7c7c7d2651571052265f1b42b4741671e44e777e76 WatchSource:0}: Error finding container 2a6513f41c418bdbfe9e1d7c7c7d2651571052265f1b42b4741671e44e777e76: Status 404 returned error can't find the container with id 2a6513f41c418bdbfe9e1d7c7c7d2651571052265f1b42b4741671e44e777e76
Nov 03 23:45:34.570404 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.570356    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:34.570515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.570454    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:34.570515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.570502    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:45:34.710455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.709999    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerStarted","Data":"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"}
Nov 03 23:45:34.712134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.712099    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerStarted","Data":"2a6513f41c418bdbfe9e1d7c7c7d2651571052265f1b42b4741671e44e777e76"}
Nov 03 23:45:34.746498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:34.745223    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podStartSLOduration=7.745198352 podStartE2EDuration="7.745198352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:34.734613547 +0000 UTC m=+950.625675482" watchObservedRunningTime="2026-11-03 23:45:34.745198352 +0000 UTC m=+950.636260288"
Nov 03 23:45:34.836129 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:45:34.836040    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:45:35.030558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.030526    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:35.035621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.035591    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:35.722304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.722267    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerStarted","Data":"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"}
Nov 03 23:45:35.723025 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.722992    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:35.723934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.723914    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:45:35.753936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:35.753883    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podStartSLOduration=3.753866297 podStartE2EDuration="3.753866297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:35.75084263 +0000 UTC m=+951.641904565" watchObservedRunningTime="2026-11-03 23:45:35.753866297 +0000 UTC m=+951.644928209"
Nov 03 23:45:36.240981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:36.240940    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:36.247285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:36.247251    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:36.729298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:36.729262    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:36.730266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:36.730246    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:45:37.737725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:37.737684    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" event={"ID":"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2","Type":"ContainerStarted","Data":"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"}
Nov 03 23:45:37.738166 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:37.737784    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:37.759154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:37.759104    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" podStartSLOduration=1.942541668 podStartE2EDuration="15.759091015s" totalImagesPullingTime="13.816549347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:37.758114616 +0000 UTC m=+953.649176552" watchObservedRunningTime="2026-11-03 23:45:37.759091015 +0000 UTC m=+953.650152947"
Nov 03 23:45:42.368132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.368093    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:45:42.376580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.376549    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.384722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.384656    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-344261-serving-cert\""
Nov 03 23:45:42.385492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.385466    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:45:42.537466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.523177    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.538682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.538635    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.542825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.538992    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gbh\" (UniqueName: \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.543075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.543050    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.645170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.645077    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.645170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.645123    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.645362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.645177    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gbh\" (UniqueName: \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.645362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.645198    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.645868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.645841    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.647781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.647759    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.647909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.647825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.658847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.658818    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gbh\" (UniqueName: \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\") pod \"maas-api-e2e-api-a-344261-897b9584d-z6bzb\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.696976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.696935    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:42.981675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:42.981643    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:45:42.982506 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:42.982473    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0132282c_1ceb_4dcd_94c0_9e3db984eaef.slice/crio-1efe7b788d57a5dc3a96ef98defb1baef9e87098e14c9d7e13e6ab936d70d2f8 WatchSource:0}: Error finding container 1efe7b788d57a5dc3a96ef98defb1baef9e87098e14c9d7e13e6ab936d70d2f8: Status 404 returned error can't find the container with id 1efe7b788d57a5dc3a96ef98defb1baef9e87098e14c9d7e13e6ab936d70d2f8
Nov 03 23:45:43.848453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:43.848412    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" event={"ID":"0132282c-1ceb-4dcd-94c0-9e3db984eaef","Type":"ContainerStarted","Data":"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"}
Nov 03 23:45:43.848917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:43.848471    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" event={"ID":"0132282c-1ceb-4dcd-94c0-9e3db984eaef","Type":"ContainerStarted","Data":"1efe7b788d57a5dc3a96ef98defb1baef9e87098e14c9d7e13e6ab936d70d2f8"}
Nov 03 23:45:43.848917 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:43.848491    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:43.873381 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:43.873323    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" podStartSLOduration=1.3541233830000001 podStartE2EDuration="1.873309754s" totalImagesPullingTime="519.186371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:43.870776059 +0000 UTC m=+959.761837995" watchObservedRunningTime="2026-11-03 23:45:43.873309754 +0000 UTC m=+959.764371696"
Nov 03 23:45:48.792408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:48.792319    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:45:48.847773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:48.847729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:45:49.414714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.414676    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:45:49.420191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.420163    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.429875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.429849    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:45:49.432112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.432089    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-953960\""
Nov 03 23:45:49.433902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.433882    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-953960-dockercfg-rc6ng\""
Nov 03 23:45:49.448355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.448319    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:45:49.453552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.453523    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.464840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.464808    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:45:49.490268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.490234    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qpz\" (UniqueName: \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.490436 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.490352    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.490481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.490457    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmc4w\" (UniqueName: \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.490524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.490481    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.617418    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qpz\" (UniqueName: \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.617545    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.617630    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc4w\" (UniqueName: \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.617663    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.618854    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.629666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.619945    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.659344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.658936    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qpz\" (UniqueName: \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\") pod \"payload-processing-e2e-api-a-953960-55bc7fccc8-sthck\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") " pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.701466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.675477    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc4w\" (UniqueName: \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\") pod \"payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.743422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.737672    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:49.790096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.789971    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:49.873883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:49.871595    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:45:50.325687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:50.325659    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:45:50.326575 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:50.326551    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec7fad9_4c78_4eae_8be3_cb2225fb56cd.slice/crio-5ee367172dc5774a89e8961cf7d9f4966c2535b7c5cf97ff80bb27434c2b5ad9 WatchSource:0}: Error finding container 5ee367172dc5774a89e8961cf7d9f4966c2535b7c5cf97ff80bb27434c2b5ad9: Status 404 returned error can't find the container with id 5ee367172dc5774a89e8961cf7d9f4966c2535b7c5cf97ff80bb27434c2b5ad9
Nov 03 23:45:50.339643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:50.339620    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:45:50.341055 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:50.341030    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48f41c3_de2d_4d7e_af93_d8da0d06828f.slice/crio-f217913ed33b37490767703500d17d020255c0a4531e8a7ec858e7f438ca8d20 WatchSource:0}: Error finding container f217913ed33b37490767703500d17d020255c0a4531e8a7ec858e7f438ca8d20: Status 404 returned error can't find the container with id f217913ed33b37490767703500d17d020255c0a4531e8a7ec858e7f438ca8d20
Nov 03 23:45:51.034482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.033297    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" event={"ID":"f48f41c3-de2d-4d7e-af93-d8da0d06828f","Type":"ContainerStarted","Data":"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"}
Nov 03 23:45:51.034482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.033367    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" event={"ID":"f48f41c3-de2d-4d7e-af93-d8da0d06828f","Type":"ContainerStarted","Data":"f217913ed33b37490767703500d17d020255c0a4531e8a7ec858e7f438ca8d20"}
Nov 03 23:45:51.034482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.033434    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:45:51.039452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.036179    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" event={"ID":"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd","Type":"ContainerStarted","Data":"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"}
Nov 03 23:45:51.039452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.036238    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" event={"ID":"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd","Type":"ContainerStarted","Data":"5ee367172dc5774a89e8961cf7d9f4966c2535b7c5cf97ff80bb27434c2b5ad9"}
Nov 03 23:45:51.039452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.036954    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:45:51.103793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.103735    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" podStartSLOduration=2.103721206 podStartE2EDuration="2.103721206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:51.102505347 +0000 UTC m=+966.993567281" watchObservedRunningTime="2026-11-03 23:45:51.103721206 +0000 UTC m=+966.994783141"
Nov 03 23:45:51.104227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:51.104197    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" podStartSLOduration=2.104189355 podStartE2EDuration="2.104189355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:51.0728793 +0000 UTC m=+966.963941236" watchObservedRunningTime="2026-11-03 23:45:51.104189355 +0000 UTC m=+966.995251309"
Nov 03 23:45:57.734153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.734116    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:45:57.738544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.738525    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:57.748987 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.748961    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc7bd0\""
Nov 03 23:45:57.749126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.749085    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cc7bd0-dockercfg-n8dr8\""
Nov 03 23:45:57.750715 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.750688    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:45:57.787859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.787824    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:45:57.792210 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.792186    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:57.802112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.802082    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:45:57.991473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.989718    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9bg\" (UniqueName: \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:57.991473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:57.990020    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.095428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.095298    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87j2n\" (UniqueName: \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.095618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.095382    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.095618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.095597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.095747 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.095688    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9bg\" (UniqueName: \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.100122 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.097068    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.121420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.117132    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9bg\" (UniqueName: \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\") pod \"payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.203417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.197163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87j2n\" (UniqueName: \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.203417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.197356    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.203417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.198438    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.249472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.247365    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87j2n\" (UniqueName: \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\") pod \"payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.355993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.355956    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:58.414232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.414136    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:58.701881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.701858    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:45:58.703045 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:58.703019    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124a50ab_fb9b_49d0_aa06_4a6986c3d908.slice/crio-2b10f2cdce406adfaf9fc3558db48ae96b6db6ae6290d3e3425f17873860ef6b WatchSource:0}: Error finding container 2b10f2cdce406adfaf9fc3558db48ae96b6db6ae6290d3e3425f17873860ef6b: Status 404 returned error can't find the container with id 2b10f2cdce406adfaf9fc3558db48ae96b6db6ae6290d3e3425f17873860ef6b
Nov 03 23:45:58.743961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:45:58.743929    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc936f359_9af4_483e_bb97_bd4b0c687a0a.slice/crio-938a72aca244ecdfad01a072c2514cc49cdeaea1c34ba10f9587acbfd041e2e9 WatchSource:0}: Error finding container 938a72aca244ecdfad01a072c2514cc49cdeaea1c34ba10f9587acbfd041e2e9: Status 404 returned error can't find the container with id 938a72aca244ecdfad01a072c2514cc49cdeaea1c34ba10f9587acbfd041e2e9
Nov 03 23:45:58.744893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:58.744858    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:45:59.237680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.237555    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" event={"ID":"c936f359-9af4-483e-bb97-bd4b0c687a0a","Type":"ContainerStarted","Data":"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"}
Nov 03 23:45:59.237680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.237634    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" event={"ID":"c936f359-9af4-483e-bb97-bd4b0c687a0a","Type":"ContainerStarted","Data":"938a72aca244ecdfad01a072c2514cc49cdeaea1c34ba10f9587acbfd041e2e9"}
Nov 03 23:45:59.238315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.238298    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:45:59.246426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.242676    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" event={"ID":"124a50ab-fb9b-49d0-aa06-4a6986c3d908","Type":"ContainerStarted","Data":"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"}
Nov 03 23:45:59.246426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.242736    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" event={"ID":"124a50ab-fb9b-49d0-aa06-4a6986c3d908","Type":"ContainerStarted","Data":"2b10f2cdce406adfaf9fc3558db48ae96b6db6ae6290d3e3425f17873860ef6b"}
Nov 03 23:45:59.246426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.243615    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:45:59.298315 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.298259    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" podStartSLOduration=2.298243514 podStartE2EDuration="2.298243514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:59.264001196 +0000 UTC m=+975.155063131" watchObservedRunningTime="2026-11-03 23:45:59.298243514 +0000 UTC m=+975.189305463"
Nov 03 23:45:59.312110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.312049    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" podStartSLOduration=2.312030459 podStartE2EDuration="2.312030459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:59.311160213 +0000 UTC m=+975.202222147" watchObservedRunningTime="2026-11-03 23:45:59.312030459 +0000 UTC m=+975.203092396"
Nov 03 23:45:59.475417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.467519    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 03 23:45:59.562083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.478642    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.582498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.581484    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 03 23:45:59.654131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654088    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654283 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654218    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2f6\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654283 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654251    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654397 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654365    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654469    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654513    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654589    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654637 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654620    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.654667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.654656    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.755981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.755940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756003    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756037    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756287    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756361    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2f6\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756412    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756451    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756472    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756474    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756556    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.756920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.756838    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.757089 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.757071    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.758548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.758521    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.758751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.758733    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.772184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.772157    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.772284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.772265    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2f6\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:45:59.810254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:45:59.810222    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:46:00.099829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.099802    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 03 23:46:00.102487 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:46:00.102456    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda655dc6a_85fc_4e2a_b956_57020d714d37.slice/crio-d59c8402a385d01c0dd4d0e23bd85d4a50f58d1fef8ebf655dee48674c882ee7 WatchSource:0}: Error finding container d59c8402a385d01c0dd4d0e23bd85d4a50f58d1fef8ebf655dee48674c882ee7: Status 404 returned error can't find the container with id d59c8402a385d01c0dd4d0e23bd85d4a50f58d1fef8ebf655dee48674c882ee7
Nov 03 23:46:00.105525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.105492    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:00.105598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.105560    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:00.105598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.105586    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:00.253480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.253441    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" event={"ID":"a655dc6a-85fc-4e2a-b956-57020d714d37","Type":"ContainerStarted","Data":"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"}
Nov 03 23:46:00.253748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.253506    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" event={"ID":"a655dc6a-85fc-4e2a-b956-57020d714d37","Type":"ContainerStarted","Data":"d59c8402a385d01c0dd4d0e23bd85d4a50f58d1fef8ebf655dee48674c882ee7"}
Nov 03 23:46:00.286567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.286506    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podStartSLOduration=1.28648848 podStartE2EDuration="1.28648848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:00.279888627 +0000 UTC m=+976.170950566" watchObservedRunningTime="2026-11-03 23:46:00.28648848 +0000 UTC m=+976.177550419"
Nov 03 23:46:00.823418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:00.813480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:46:01.831634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:01.831588    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:01.832098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:01.831684    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:02.066338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:02.066306    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:46:02.066934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:02.066909    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:46:02.818410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:02.812466    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:02.818410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:02.812534    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:03.828445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:03.810934    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:03.828445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:03.811009    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:04.811111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:04.811078    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:04.811453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:04.811419    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:04.989082 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:04.989048    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:46:05.829414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:05.825869    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:05.829414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:05.825936    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:06.814420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:06.813799    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 23:46:06.814420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:06.813880    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 23:46:07.882451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:07.879190    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:46:07.882451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:07.879686    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:46:07.882451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:07.881125    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 03 23:46:09.235870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.234652    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:46:09.257567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.256982    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.262528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.262496    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:46:09.269871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.269838    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-32de63-openshift-default-dockercfg-dv2t6\""
Nov 03 23:46:09.423224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.423175    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.428633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.428597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.428873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.428846    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.433071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.433035    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6k4j\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.435378 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.435343    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.435758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.435732    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.441500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.441472    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.443584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.443554    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.443844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.443826    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549372    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549496    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549552    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549655    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549765    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549827    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6k4j\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.549900    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.550030    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.550515    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.550756    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.551965    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.553700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.552192    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.566282 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.565032    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.576649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.576620    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.580740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.580262    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.580912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.580885    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6k4j\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.587564 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.587533    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\") pod \"e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") " pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:09.878955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:09.878873    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:10.256598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.256525    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:46:10.276034 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.276005    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:46:10.277832 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:46:10.277803    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cad2477_b528_47df_9d1d_68830d853cf8.slice/crio-bb7bb4553a51ee6d527161f4856864ff1eebf631d0894e5182b381b27e72262c WatchSource:0}: Error finding container bb7bb4553a51ee6d527161f4856864ff1eebf631d0894e5182b381b27e72262c: Status 404 returned error can't find the container with id bb7bb4553a51ee6d527161f4856864ff1eebf631d0894e5182b381b27e72262c
Nov 03 23:46:10.281044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.281003    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:10.281129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.281093    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:10.281176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.281137    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:10.465849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.465813    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" event={"ID":"8cad2477-b528-47df-9d1d-68830d853cf8","Type":"ContainerStarted","Data":"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"}
Nov 03 23:46:10.466019 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.465866    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" event={"ID":"8cad2477-b528-47df-9d1d-68830d853cf8","Type":"ContainerStarted","Data":"bb7bb4553a51ee6d527161f4856864ff1eebf631d0894e5182b381b27e72262c"}
Nov 03 23:46:10.496040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.495971    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" podStartSLOduration=1.495950307 podStartE2EDuration="1.495950307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:10.490550055 +0000 UTC m=+986.381611992" watchObservedRunningTime="2026-11-03 23:46:10.495950307 +0000 UTC m=+986.387012243"
Nov 03 23:46:10.879642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:10.879611    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:11.273069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:11.272984    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:46:11.885251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:11.885217    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:12.592413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:12.587321    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:12.613362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:12.604810    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:46:14.581182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.581146    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:46:14.587147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.587104    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:46:14.587822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.587793    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:14.594102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.593823    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:14.605815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.597888    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-32de63\""
Nov 03 23:46:14.605815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.598017    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:46:14.605815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.600114    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:46:14.605815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.604144    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-32de63-dockercfg-6zxzs\""
Nov 03 23:46:14.686514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.686479    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jgf\" (UniqueName: \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:14.686760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.686745    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295zv\" (UniqueName: \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:14.686892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.686878    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:14.687203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.687190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:14.877081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.876984    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:15.013411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.012140    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jgf\" (UniqueName: \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:15.013411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.012250    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295zv\" (UniqueName: \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:15.013411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.012327    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:15.013859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.013757    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:15.013859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:14.917001    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:15.093957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.090163    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jgf\" (UniqueName: \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\") pod \"payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:15.194530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.187719    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295zv\" (UniqueName: \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\") pod \"payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:15.298413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.295110    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:15.304540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:15.304504    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:16.371150 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:16.371123    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:46:16.373355 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:46:16.373325    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b39b8af_d182_45b5_a404_ed6cdccfee66.slice/crio-bab0d3f5878559525ccdae1ecea6835ad28c21e219170e200ceb88103a5204ee WatchSource:0}: Error finding container bab0d3f5878559525ccdae1ecea6835ad28c21e219170e200ceb88103a5204ee: Status 404 returned error can't find the container with id bab0d3f5878559525ccdae1ecea6835ad28c21e219170e200ceb88103a5204ee
Nov 03 23:46:16.461413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:16.459724    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:46:17.384563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.384497    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" event={"ID":"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62","Type":"ContainerStarted","Data":"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"}
Nov 03 23:46:17.385037 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.384620    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" event={"ID":"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62","Type":"ContainerStarted","Data":"c0b08102b7521045b2231b3f923981b087cb216fd1227604da7aff6bb38624d8"}
Nov 03 23:46:17.385037 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.384669    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:17.386735 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.386706    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" event={"ID":"7b39b8af-d182-45b5-a404-ed6cdccfee66","Type":"ContainerStarted","Data":"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"}
Nov 03 23:46:17.386872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.386752    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" event={"ID":"7b39b8af-d182-45b5-a404-ed6cdccfee66","Type":"ContainerStarted","Data":"bab0d3f5878559525ccdae1ecea6835ad28c21e219170e200ceb88103a5204ee"}
Nov 03 23:46:17.386946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.386868    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:17.431151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.430150    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" podStartSLOduration=3.430130589 podStartE2EDuration="3.430130589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:17.40614748 +0000 UTC m=+993.297209415" watchObservedRunningTime="2026-11-03 23:46:17.430130589 +0000 UTC m=+993.321192529"
Nov 03 23:46:17.463463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:17.456950    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" podStartSLOduration=3.456932252 podStartE2EDuration="3.456932252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:17.432787038 +0000 UTC m=+993.323848984" watchObservedRunningTime="2026-11-03 23:46:17.456932252 +0000 UTC m=+993.347994188"
Nov 03 23:46:21.767270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:21.767222    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:46:21.767901 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:21.767323    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:46:22.375874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.375842    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:46:22.382682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.382652    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.388705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.388674    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:46:22.391741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.391715    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-32de63-serving-cert\""
Nov 03 23:46:22.504186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.504152    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.504483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.504213    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.504483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.504375    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqsn\" (UniqueName: \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.504483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.504448    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.608473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.606998    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.608473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.607103    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.608473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.607205    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqsn\" (UniqueName: \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.608473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.607269    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.613528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.612820    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.637583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.637099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.660020 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.659985    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.663690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.663657    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqsn\" (UniqueName: \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\") pod \"maas-api-e2e-api-b-32de63-699c5dc858-xc7gx\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.718962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.718686    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:22.781721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.781688    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:46:22.782115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:22.781741    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:46:23.171947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:23.171839    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:46:23.446592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:23.446507    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" event={"ID":"ede29a37-da72-4da3-bc1f-83087e266934","Type":"ContainerStarted","Data":"4fd98a98926cca9d7723451e8a0656c34e6cbe63ab50e7acbd5869019ccf8f4c"}
Nov 03 23:46:24.458933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:24.458882    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" event={"ID":"ede29a37-da72-4da3-bc1f-83087e266934","Type":"ContainerStarted","Data":"9148f01110b69d0c2b3832ed9b1d909e06776b40760862dcdf90871d0a79de3d"}
Nov 03 23:46:24.459433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:24.458967    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:24.485244 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:24.485166    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" podStartSLOduration=1.852765701 podStartE2EDuration="2.485147168s" totalImagesPullingTime="632.381467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:24.482436541 +0000 UTC m=+1000.373498470" watchObservedRunningTime="2026-11-03 23:46:24.485147168 +0000 UTC m=+1000.376209104"
Nov 03 23:46:25.479367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.479326    6430 generic.go:356] "Generic (PLEG): container finished" podID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerID="81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6" exitCode=0
Nov 03 23:46:25.480615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.479442    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerDied","Data":"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"}
Nov 03 23:46:25.480615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.480049    6430 scope.go:122] "RemoveContainer" containerID="81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"
Nov 03 23:46:25.481719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.481690    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:25.481792 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.481766    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:25.481830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:25.481796    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:26.489151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.489109    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerStarted","Data":"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"}
Nov 03 23:46:26.490767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.490747    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerID="79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7" exitCode=0
Nov 03 23:46:26.490839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.490813    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerDied","Data":"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"}
Nov 03 23:46:26.491151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.491136    6430 scope.go:122] "RemoveContainer" containerID="79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"
Nov 03 23:46:26.492442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.492418    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:26.492496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.492482    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:26.492530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:26.492514    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:27.241186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:27.241149    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:27.241429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:27.241206    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:27.241429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:27.241253    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:27.500896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:27.500809    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerStarted","Data":"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"}
Nov 03 23:46:28.030646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.030556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:46:28.031232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.031206    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:28.031446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.031422    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:28.245412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.244529    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:28.245412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.244587    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:28.399372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.399340    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:46:28.403806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.403784    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:46:28.404848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.404820    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:46:28.405050 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:28.405025    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" podUID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" containerName="maas-api" containerID="cri-o://a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da" gracePeriod=30
Nov 03 23:46:29.056418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.048877    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:29.056418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.048936    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:29.289599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.289517    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:29.289847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.289815    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:29.397112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.397083    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:46:29.488540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.488511    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\") pod \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") "
Nov 03 23:46:29.488701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.488549    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\") pod \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") "
Nov 03 23:46:29.488701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.488584    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\" (UniqueName: \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\") pod \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") "
Nov 03 23:46:29.488791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.488713    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\") pod \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\" (UID: \"0132282c-1ceb-4dcd-94c0-9e3db984eaef\") "
Nov 03 23:46:29.488889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.488868    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca" pod "0132282c-1ceb-4dcd-94c0-9e3db984eaef" (UID: "0132282c-1ceb-4dcd-94c0-9e3db984eaef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:29.489552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.489525    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0132282c-1ceb-4dcd-94c0-9e3db984eaef-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:29.490807 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.490781    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs" pod "0132282c-1ceb-4dcd-94c0-9e3db984eaef" (UID: "0132282c-1ceb-4dcd-94c0-9e3db984eaef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:46:29.490912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.490815    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh" pod "0132282c-1ceb-4dcd-94c0-9e3db984eaef" (UID: "0132282c-1ceb-4dcd-94c0-9e3db984eaef"). InnerVolumeSpecName "kube-api-access-l6gbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:29.490912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.490843    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls" pod "0132282c-1ceb-4dcd-94c0-9e3db984eaef" (UID: "0132282c-1ceb-4dcd-94c0-9e3db984eaef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:46:29.583701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.583667    6430 generic.go:356] "Generic (PLEG): container finished" podID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" containerID="a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da" exitCode=0
Nov 03 23:46:29.583973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.583957    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" event={"ID":"0132282c-1ceb-4dcd-94c0-9e3db984eaef","Type":"ContainerDied","Data":"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"}
Nov 03 23:46:29.584087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.584076    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb" event={"ID":"0132282c-1ceb-4dcd-94c0-9e3db984eaef","Type":"ContainerDied","Data":"1efe7b788d57a5dc3a96ef98defb1baef9e87098e14c9d7e13e6ab936d70d2f8"}
Nov 03 23:46:29.584204 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.584193    6430 scope.go:122] "RemoveContainer" containerID="a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"
Nov 03 23:46:29.584456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.584441    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"
Nov 03 23:46:29.591788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.591748    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:29.591929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.591911    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gbh\" (UniqueName: \"kubernetes.io/projected/0132282c-1ceb-4dcd-94c0-9e3db984eaef-kube-api-access-l6gbh\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:29.592002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.591936    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0132282c-1ceb-4dcd-94c0-9e3db984eaef-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:29.601637 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.601622    6430 scope.go:122] "RemoveContainer" containerID="a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"
Nov 03 23:46:29.601880 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:29.601858    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da\": container with ID starting with a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da not found: ID does not exist" containerID="a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"
Nov 03 23:46:29.601929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.601889    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da"} err="failed to get container status \"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da\": rpc error: code = NotFound desc = could not find container \"a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da\": container with ID starting with a674a659415945a24ffa0327397da8f48011cc17750688c6df661e2edb7995da not found: ID does not exist"
Nov 03 23:46:29.651405 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.651346    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:46:29.665443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:29.665407    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-344261-897b9584d-z6bzb"]
Nov 03 23:46:30.048484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.048215    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:30.048484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.048284    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:30.261422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.260666    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:30.261422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.260722    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:30.268191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.267946    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" path="/var/lib/kubelet/pods/0132282c-1ceb-4dcd-94c0-9e3db984eaef/volumes"
Nov 03 23:46:30.485058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:30.484983    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:46:31.033827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:31.033786    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:31.034110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:31.034075    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:31.244411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:31.243583    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:31.244411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:31.243641    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:32.042414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.035883    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:32.042414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.036035    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:32.273411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.253242    6430 patch_prober.go:29] interesting pod/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 23:46:32.273411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.253418    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 23:46:32.768881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.768846    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:46:32.769548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.769530    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" containerName="maas-api"
Nov 03 23:46:32.769647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.769548    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" containerName="maas-api"
Nov 03 23:46:32.769729 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.769713    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0132282c-1ceb-4dcd-94c0-9e3db984eaef" containerName="maas-api"
Nov 03 23:46:32.775380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.775362    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.786420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.786382    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88f34dbe-openshift-default-dockercfg-nhpxr\""
Nov 03 23:46:32.790271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.790249    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:46:32.941509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941468    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.941704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941560    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.941704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941698    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.941813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941758    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.941865 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941853    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rff2\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.941969 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.941942    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.942111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.942089    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.942300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.942123    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:32.942300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:32.942243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.033351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.032833    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 23:46:33.033351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.032893    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 23:46:33.043590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043701    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043766    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043855    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043925    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rff2\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.043969    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044056    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044106    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044343    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044576    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044591    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.044861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.044839    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.046529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.046506    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.046789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.046770    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.081409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.078940    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rff2\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.081409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.079345    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.092494 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.092451    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:33.255258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.255226    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:33.256346 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.256323    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:33.257437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.257280    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:33.370363 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.369895    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:46:33.599495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.599455    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:46:33.659136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.659085    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:33.659261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.659187    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:33.659261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:33.659234    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:46:34.701825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:34.698186    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" event={"ID":"814cf5ad-780e-49e2-96b8-453bf82698be","Type":"ContainerStarted","Data":"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"}
Nov 03 23:46:34.701825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:34.698245    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" event={"ID":"814cf5ad-780e-49e2-96b8-453bf82698be","Type":"ContainerStarted","Data":"85668b75bfd3c837ecedba2c4f8043bd8cc092fb31d07031f3bafc9e5925a40d"}
Nov 03 23:46:34.757478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:34.756744    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podStartSLOduration=2.756702954 podStartE2EDuration="2.756702954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:34.740756747 +0000 UTC m=+1010.631818681" watchObservedRunningTime="2026-11-03 23:46:34.756702954 +0000 UTC m=+1010.647764892"
Nov 03 23:46:35.141410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.073882    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:46:35.149538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.075767    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:46:35.149701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.149576    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:46:35.172412 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:35.098619    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:46:35.172412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.109347    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:35.172412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.159700    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:35.172412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.149512    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:35.314426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.253834    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:46:35.314426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.270872    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" containerID="cri-o://567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e" gracePeriod=30
Nov 03 23:46:35.314426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.271337    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:46:35.375625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.345145    6430 patch_prober.go:29] interesting pod/e2e-api-a-344261-openshift-default-85f479895c-tncch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:46:35.375625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:35.345215    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:46:36.094431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:36.094384    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:36.094836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:36.094463    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:37.094240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:37.094205    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:37.094453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:37.094257    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:38.094402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:38.094364    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:38.094570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:38.094432    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:39.108416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:39.106981    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:39.108416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:39.107059    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:39.448077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:39.447993    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:46:39.455733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:39.455700    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:46:40.094406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.094360    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:40.094581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.094435    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:40.617311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.616931    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:46:40.702626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702595    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.702790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702662    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.702790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702768    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.702902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702813    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.702902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702893    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.703002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702942    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:40.703002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.702955    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.703106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703031    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.703165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703143    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.703233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703176    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:40.703354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703315    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:40.703456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703355    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:40.703541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703455    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:40.703541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.703518    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\") pod \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\" (UID: \"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58\") "
Nov 03 23:46:40.704479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.704454    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.705138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.705109    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:46:40.705276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.705256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:40.705319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.705287    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "kube-api-access-fcdsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:40.706194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.706181    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy" pod "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" (UID: "3d3f2edb-bc84-41fe-bf85-af7b44eb6a58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:40.806108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806077    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806103    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806113    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806123    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806132    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdsr\" (UniqueName: \"kubernetes.io/projected/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-kube-api-access-fcdsr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806142    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806152    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.806320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.806161    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:40.844117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.844085    6430 generic.go:356] "Generic (PLEG): container finished" podID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerID="567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e" exitCode=0
Nov 03 23:46:40.844299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.844156    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"
Nov 03 23:46:40.844299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.844174    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerDied","Data":"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"}
Nov 03 23:46:40.844299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.844229    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch" event={"ID":"3d3f2edb-bc84-41fe-bf85-af7b44eb6a58","Type":"ContainerDied","Data":"7b54370fc078574fc2b83e9c428d3bf8cfdf9abae1aaf251be4df7852b9c7d9e"}
Nov 03 23:46:40.844299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.844254    6430 scope.go:122] "RemoveContainer" containerID="567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"
Nov 03 23:46:40.891560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.887769    6430 scope.go:122] "RemoveContainer" containerID="79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"
Nov 03 23:46:40.982441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.980593    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:46:40.995050 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.995014    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-344261-openshift-default-85f479895c-tncch"]
Nov 03 23:46:40.997464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.997444    6430 scope.go:122] "RemoveContainer" containerID="567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"
Nov 03 23:46:40.997837 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:40.997814    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e\": container with ID starting with 567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e not found: ID does not exist" containerID="567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"
Nov 03 23:46:40.997930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.997848    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e"} err="failed to get container status \"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e\": rpc error: code = NotFound desc = could not find container \"567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e\": container with ID starting with 567a927b0cfe619fded5bc0c1547b3c9f842f1b794824848db18b729a53e717e not found: ID does not exist"
Nov 03 23:46:40.997930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.997874    6430 scope.go:122] "RemoveContainer" containerID="79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"
Nov 03 23:46:40.998175 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:40.998155    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7\": container with ID starting with 79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7 not found: ID does not exist" containerID="79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"
Nov 03 23:46:40.998247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:40.998182    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7"} err="failed to get container status \"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7\": rpc error: code = NotFound desc = could not find container \"79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7\": container with ID starting with 79862cc7305a949234eeff33b8e82a4c75d4d045f58d76ad8bd95edbe03f07f7 not found: ID does not exist"
Nov 03 23:46:41.094139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:41.094107    6430 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 23:46:41.094305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:41.094159    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 23:46:41.456710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:41.456674    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:46:41.475101 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:41.475068    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:46:42.097474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:42.097444    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:42.097956 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:42.097918    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:42.098547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:42.098527    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:46:42.233218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:42.233189    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" path="/var/lib/kubelet/pods/3d3f2edb-bc84-41fe-bf85-af7b44eb6a58/volumes"
Nov 03 23:46:44.448861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.448774    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" podUID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" containerName="payload-processing" containerID="cri-o://081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669" gracePeriod=25
Nov 03 23:46:44.456522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.456492    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" podUID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" containerName="payload-pre-processing" containerID="cri-o://b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390" gracePeriod=25
Nov 03 23:46:44.743067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.743039    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:46:44.854254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.848105    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:46:44.856067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.856028    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\" (UniqueName: \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\") pod \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") "
Nov 03 23:46:44.856214 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.856118    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\") pod \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\" (UID: \"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd\") "
Nov 03 23:46:44.870911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.870868    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume" pod "6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" (UID: "6ec7fad9-4c78-4eae-8be3-cb2225fb56cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:44.874073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.874033    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz" pod "6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" (UID: "6ec7fad9-4c78-4eae-8be3-cb2225fb56cd"). InnerVolumeSpecName "kube-api-access-82qpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:44.958627 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.958583    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\") pod \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") "
Nov 03 23:46:44.958787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.958664    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\" (UniqueName: \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\") pod \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\" (UID: \"f48f41c3-de2d-4d7e-af93-d8da0d06828f\") "
Nov 03 23:46:44.959561 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.959539    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qpz\" (UniqueName: \"kubernetes.io/projected/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-kube-api-access-82qpz\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:44.959601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.959568    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:44.959790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.959767    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume" pod "f48f41c3-de2d-4d7e-af93-d8da0d06828f" (UID: "f48f41c3-de2d-4d7e-af93-d8da0d06828f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:44.961573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:44.961516    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w" pod "f48f41c3-de2d-4d7e-af93-d8da0d06828f" (UID: "f48f41c3-de2d-4d7e-af93-d8da0d06828f"). InnerVolumeSpecName "kube-api-access-fmc4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:45.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.061371    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48f41c3-de2d-4d7e-af93-d8da0d06828f-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:45.061420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.061422    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmc4w\" (UniqueName: \"kubernetes.io/projected/f48f41c3-de2d-4d7e-af93-d8da0d06828f-kube-api-access-fmc4w\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:45.157573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.157536    6430 generic.go:356] "Generic (PLEG): container finished" podID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" containerID="081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669" exitCode=0
Nov 03 23:46:45.157719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.157588    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"
Nov 03 23:46:45.157719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.157616    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" event={"ID":"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd","Type":"ContainerDied","Data":"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"}
Nov 03 23:46:45.157719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.157670    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck" event={"ID":"6ec7fad9-4c78-4eae-8be3-cb2225fb56cd","Type":"ContainerDied","Data":"5ee367172dc5774a89e8961cf7d9f4966c2535b7c5cf97ff80bb27434c2b5ad9"}
Nov 03 23:46:45.157719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.157693    6430 scope.go:122] "RemoveContainer" containerID="081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"
Nov 03 23:46:45.159257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.159218    6430 generic.go:356] "Generic (PLEG): container finished" podID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" containerID="b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390" exitCode=0
Nov 03 23:46:45.159363 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.159262    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"
Nov 03 23:46:45.159363 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.159277    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" event={"ID":"f48f41c3-de2d-4d7e-af93-d8da0d06828f","Type":"ContainerDied","Data":"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"}
Nov 03 23:46:45.159363 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.159328    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7" event={"ID":"f48f41c3-de2d-4d7e-af93-d8da0d06828f","Type":"ContainerDied","Data":"f217913ed33b37490767703500d17d020255c0a4531e8a7ec858e7f438ca8d20"}
Nov 03 23:46:45.171002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.170540    6430 scope.go:122] "RemoveContainer" containerID="081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"
Nov 03 23:46:45.171002 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:45.170896    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669\": container with ID starting with 081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669 not found: ID does not exist" containerID="081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"
Nov 03 23:46:45.171002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.170936    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669"} err="failed to get container status \"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669\": rpc error: code = NotFound desc = could not find container \"081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669\": container with ID starting with 081d30d1f4f50a14bf21678bd661a995813cda382649e83cfc847e7676bca669 not found: ID does not exist"
Nov 03 23:46:45.171002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.170959    6430 scope.go:122] "RemoveContainer" containerID="b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"
Nov 03 23:46:45.180311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.180289    6430 scope.go:122] "RemoveContainer" containerID="b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"
Nov 03 23:46:45.180611 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:45.180593    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390\": container with ID starting with b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390 not found: ID does not exist" containerID="b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"
Nov 03 23:46:45.180682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.180618    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390"} err="failed to get container status \"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390\": rpc error: code = NotFound desc = could not find container \"b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390\": container with ID starting with b9cad3e618ebbc4759c9189c5093cde09f70da88707aef946fe508d05db85390 not found: ID does not exist"
Nov 03 23:46:45.190470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.190446    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:46:45.196028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.196005    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-953960-55bc7fccc8-sthck"]
Nov 03 23:46:45.211683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.211623    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:46:45.220131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:45.220106    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-953960-644869cc9f-zkcl7"]
Nov 03 23:46:46.237061 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.237022    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" path="/var/lib/kubelet/pods/6ec7fad9-4c78-4eae-8be3-cb2225fb56cd/volumes"
Nov 03 23:46:46.237536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.237522    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" path="/var/lib/kubelet/pods/f48f41c3-de2d-4d7e-af93-d8da0d06828f/volumes"
Nov 03 23:46:46.458012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.457969    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" podUID="c936f359-9af4-483e-bb97-bd4b0c687a0a" containerName="payload-processing" containerID="cri-o://b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48" gracePeriod=25
Nov 03 23:46:46.476667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.476352    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" podUID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" containerName="payload-pre-processing" containerID="cri-o://bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41" gracePeriod=25
Nov 03 23:46:46.770466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.770432    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:46:46.807830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.807796    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\") pod \"c936f359-9af4-483e-bb97-bd4b0c687a0a\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") "
Nov 03 23:46:46.807994 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.807871    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\" (UniqueName: \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\") pod \"c936f359-9af4-483e-bb97-bd4b0c687a0a\" (UID: \"c936f359-9af4-483e-bb97-bd4b0c687a0a\") "
Nov 03 23:46:46.809033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.809007    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume" pod "c936f359-9af4-483e-bb97-bd4b0c687a0a" (UID: "c936f359-9af4-483e-bb97-bd4b0c687a0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:46.810552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.810519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n" pod "c936f359-9af4-483e-bb97-bd4b0c687a0a" (UID: "c936f359-9af4-483e-bb97-bd4b0c687a0a"). InnerVolumeSpecName "kube-api-access-87j2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:46.834006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.833972    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:46:46.908988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.908951    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\" (UniqueName: \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\") pod \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") "
Nov 03 23:46:46.909167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.909038    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\") pod \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\" (UID: \"124a50ab-fb9b-49d0-aa06-4a6986c3d908\") "
Nov 03 23:46:46.909468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.909438    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume" pod "124a50ab-fb9b-49d0-aa06-4a6986c3d908" (UID: "124a50ab-fb9b-49d0-aa06-4a6986c3d908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:46.909920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.909894    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124a50ab-fb9b-49d0-aa06-4a6986c3d908-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:46.910018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.909923    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c936f359-9af4-483e-bb97-bd4b0c687a0a-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:46.910018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.909937    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87j2n\" (UniqueName: \"kubernetes.io/projected/c936f359-9af4-483e-bb97-bd4b0c687a0a-kube-api-access-87j2n\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:46.911413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:46.911367    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg" pod "124a50ab-fb9b-49d0-aa06-4a6986c3d908" (UID: "124a50ab-fb9b-49d0-aa06-4a6986c3d908"). InnerVolumeSpecName "kube-api-access-rh9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:47.012739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.012650    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9bg\" (UniqueName: \"kubernetes.io/projected/124a50ab-fb9b-49d0-aa06-4a6986c3d908-kube-api-access-rh9bg\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:47.185455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.185355    6430 generic.go:356] "Generic (PLEG): container finished" podID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" containerID="bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41" exitCode=0
Nov 03 23:46:47.185455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.185436    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"
Nov 03 23:46:47.185455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.185438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" event={"ID":"124a50ab-fb9b-49d0-aa06-4a6986c3d908","Type":"ContainerDied","Data":"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"}
Nov 03 23:46:47.185723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.185505    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j" event={"ID":"124a50ab-fb9b-49d0-aa06-4a6986c3d908","Type":"ContainerDied","Data":"2b10f2cdce406adfaf9fc3558db48ae96b6db6ae6290d3e3425f17873860ef6b"}
Nov 03 23:46:47.185723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.185536    6430 scope.go:122] "RemoveContainer" containerID="bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"
Nov 03 23:46:47.187155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.187131    6430 generic.go:356] "Generic (PLEG): container finished" podID="c936f359-9af4-483e-bb97-bd4b0c687a0a" containerID="b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48" exitCode=0
Nov 03 23:46:47.187279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.187180    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"
Nov 03 23:46:47.187279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.187226    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" event={"ID":"c936f359-9af4-483e-bb97-bd4b0c687a0a","Type":"ContainerDied","Data":"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"}
Nov 03 23:46:47.187279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.187274    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88" event={"ID":"c936f359-9af4-483e-bb97-bd4b0c687a0a","Type":"ContainerDied","Data":"938a72aca244ecdfad01a072c2514cc49cdeaea1c34ba10f9587acbfd041e2e9"}
Nov 03 23:46:47.197110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.197089    6430 scope.go:122] "RemoveContainer" containerID="bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"
Nov 03 23:46:47.197452 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:47.197425    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41\": container with ID starting with bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41 not found: ID does not exist" containerID="bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"
Nov 03 23:46:47.197533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.197462    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41"} err="failed to get container status \"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41\": rpc error: code = NotFound desc = could not find container \"bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41\": container with ID starting with bc67d744967c63f7533cd7f6d806df99e27c3dcee6ff0dc695610ecf79975e41 not found: ID does not exist"
Nov 03 23:46:47.197533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.197507    6430 scope.go:122] "RemoveContainer" containerID="b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"
Nov 03 23:46:47.208876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.208859    6430 scope.go:122] "RemoveContainer" containerID="b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"
Nov 03 23:46:47.209169 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:47.209149    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48\": container with ID starting with b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48 not found: ID does not exist" containerID="b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"
Nov 03 23:46:47.209252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.209177    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48"} err="failed to get container status \"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48\": rpc error: code = NotFound desc = could not find container \"b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48\": container with ID starting with b5f423431fc18bbc1d808443196c23d9e579340052f44043c40f07b8db312f48 not found: ID does not exist"
Nov 03 23:46:47.253492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.253453    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:46:47.283550 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.283517    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc7bd0-784494d5bc-ftl7j"]
Nov 03 23:46:47.315056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.315002    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:46:47.323977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:47.323942    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc7bd0-64df677bc4-ptw88"]
Nov 03 23:46:48.232629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:48.232593    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" path="/var/lib/kubelet/pods/124a50ab-fb9b-49d0-aa06-4a6986c3d908/volumes"
Nov 03 23:46:48.233164 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:48.233141    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c936f359-9af4-483e-bb97-bd4b0c687a0a" path="/var/lib/kubelet/pods/c936f359-9af4-483e-bb97-bd4b0c687a0a/volumes"
Nov 03 23:46:50.176755 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:50.176714    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:46:50.177234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:50.177096    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy" containerID="cri-o://99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e" gracePeriod=30
Nov 03 23:46:55.362080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.357022    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:46:55.583787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.583759    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:55.619867 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.619833    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.619902    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.619988    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620054    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620111    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620209    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620239    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620294    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620474    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\") pod \"3c9526b9-75de-4674-9b33-1af6ee23abfa\" (UID: \"3c9526b9-75de-4674-9b33-1af6ee23abfa\") "
Nov 03 23:46:55.620557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620501    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:55.620664 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:55.620754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620694    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:55.620754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.620737    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:55.621045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621023    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:46:55.621246 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621230    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.621325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621252    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.621325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621266    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.621325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621282    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9526b9-75de-4674-9b33-1af6ee23abfa-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.621325 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.621295    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.622922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.622893    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:46:55.623172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.623144    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "kube-api-access-r9rmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:55.627651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.627627    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:46:55.627786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.627648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token" pod "3c9526b9-75de-4674-9b33-1af6ee23abfa" (UID: "3c9526b9-75de-4674-9b33-1af6ee23abfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:46:55.722349 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.722316    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.722349 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.722344    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.722633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.722359    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rmb\" (UniqueName: \"kubernetes.io/projected/3c9526b9-75de-4674-9b33-1af6ee23abfa-kube-api-access-r9rmb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:55.722633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:55.722374    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9526b9-75de-4674-9b33-1af6ee23abfa-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:46:56.254182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.254154    6430 generic.go:356] "Generic (PLEG): container finished" podID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerID="99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e" exitCode=0
Nov 03 23:46:56.254340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.254204    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"
Nov 03 23:46:56.254340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.254238    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerDied","Data":"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"}
Nov 03 23:46:56.254340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.254292    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf" event={"ID":"3c9526b9-75de-4674-9b33-1af6ee23abfa","Type":"ContainerDied","Data":"2a6513f41c418bdbfe9e1d7c7c7d2651571052265f1b42b4741671e44e777e76"}
Nov 03 23:46:56.254340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.254314    6430 scope.go:122] "RemoveContainer" containerID="99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"
Nov 03 23:46:56.263965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.263947    6430 scope.go:122] "RemoveContainer" containerID="81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"
Nov 03 23:46:56.272924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.272905    6430 scope.go:122] "RemoveContainer" containerID="99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"
Nov 03 23:46:56.273186 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:56.273169    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e\": container with ID starting with 99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e not found: ID does not exist" containerID="99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"
Nov 03 23:46:56.273249 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.273194    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e"} err="failed to get container status \"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e\": rpc error: code = NotFound desc = could not find container \"99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e\": container with ID starting with 99a87c4856fccbdcb1554ebb4be84450519bba6267c0b64f54124e12fe7d564e not found: ID does not exist"
Nov 03 23:46:56.273249 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.273210    6430 scope.go:122] "RemoveContainer" containerID="81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"
Nov 03 23:46:56.273455 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:46:56.273439    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6\": container with ID starting with 81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6 not found: ID does not exist" containerID="81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"
Nov 03 23:46:56.273508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.273460    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6"} err="failed to get container status \"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6\": rpc error: code = NotFound desc = could not find container \"81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6\": container with ID starting with 81862ca1890294eaf8bb280921970b4cfcd08d5e7745b493d1f22a4bc70ed2a6 not found: ID does not exist"
Nov 03 23:46:56.285883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.285858    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:46:56.289069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:56.289038    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-953960-openshift-default-7545d8768f-qbqbf"]
Nov 03 23:46:58.327418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:58.326754    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" path="/var/lib/kubelet/pods/3c9526b9-75de-4674-9b33-1af6ee23abfa/volumes"
Nov 03 23:46:59.785414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:59.784457    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:46:59.831023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:46:59.830984    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:47:00.212421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:00.211494    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:47:03.040795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.040755    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:47:03.041671 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.041632    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" podUID="ede29a37-da72-4da3-bc1f-83087e266934" containerName="maas-api" containerID="cri-o://9148f01110b69d0c2b3832ed9b1d909e06776b40760862dcdf90871d0a79de3d" gracePeriod=30
Nov 03 23:47:03.660880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.650472    6430 generic.go:356] "Generic (PLEG): container finished" podID="ede29a37-da72-4da3-bc1f-83087e266934" containerID="9148f01110b69d0c2b3832ed9b1d909e06776b40760862dcdf90871d0a79de3d" exitCode=0
Nov 03 23:47:03.660880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.650554    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" event={"ID":"ede29a37-da72-4da3-bc1f-83087e266934","Type":"ContainerDied","Data":"9148f01110b69d0c2b3832ed9b1d909e06776b40760862dcdf90871d0a79de3d"}
Nov 03 23:47:03.792603 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.792573    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:47:03.839285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.839246    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\") pod \"ede29a37-da72-4da3-bc1f-83087e266934\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") "
Nov 03 23:47:03.839643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.839618    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\") pod \"ede29a37-da72-4da3-bc1f-83087e266934\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") "
Nov 03 23:47:03.843726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.843701    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs" pod "ede29a37-da72-4da3-bc1f-83087e266934" (UID: "ede29a37-da72-4da3-bc1f-83087e266934"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:03.844219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.844197    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls" pod "ede29a37-da72-4da3-bc1f-83087e266934" (UID: "ede29a37-da72-4da3-bc1f-83087e266934"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:47:03.943649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.943547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\" (UniqueName: \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\") pod \"ede29a37-da72-4da3-bc1f-83087e266934\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") "
Nov 03 23:47:03.946653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.944163    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\") pod \"ede29a37-da72-4da3-bc1f-83087e266934\" (UID: \"ede29a37-da72-4da3-bc1f-83087e266934\") "
Nov 03 23:47:03.946653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.944924    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:03.946653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.944950    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede29a37-da72-4da3-bc1f-83087e266934-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:03.946653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.945353    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca" pod "ede29a37-da72-4da3-bc1f-83087e266934" (UID: "ede29a37-da72-4da3-bc1f-83087e266934"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:03.950780 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:03.950751    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn" pod "ede29a37-da72-4da3-bc1f-83087e266934" (UID: "ede29a37-da72-4da3-bc1f-83087e266934"). InnerVolumeSpecName "kube-api-access-txqsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:04.046504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.046465    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txqsn\" (UniqueName: \"kubernetes.io/projected/ede29a37-da72-4da3-bc1f-83087e266934-kube-api-access-txqsn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:04.046504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.046500    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede29a37-da72-4da3-bc1f-83087e266934-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:04.658278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.658247    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"
Nov 03 23:47:04.658512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.658284    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx" event={"ID":"ede29a37-da72-4da3-bc1f-83087e266934","Type":"ContainerDied","Data":"4fd98a98926cca9d7723451e8a0656c34e6cbe63ab50e7acbd5869019ccf8f4c"}
Nov 03 23:47:04.658512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.658360    6430 scope.go:122] "RemoveContainer" containerID="9148f01110b69d0c2b3832ed9b1d909e06776b40760862dcdf90871d0a79de3d"
Nov 03 23:47:04.682028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.681989    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:47:04.686367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.686343    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de63-699c5dc858-xc7gx"]
Nov 03 23:47:04.786331 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.786264    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" podUID="7b39b8af-d182-45b5-a404-ed6cdccfee66" containerName="payload-processing" containerID="cri-o://6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1" gracePeriod=25
Nov 03 23:47:04.832870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:04.832811    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" podUID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" containerName="payload-pre-processing" containerID="cri-o://9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd" gracePeriod=25
Nov 03 23:47:04.859486 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:04.859429    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:47:05.342750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.342717    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:47:05.398777 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.398746    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\" (UniqueName: \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\") pod \"7b39b8af-d182-45b5-a404-ed6cdccfee66\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") "
Nov 03 23:47:05.398946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.398870    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\") pod \"7b39b8af-d182-45b5-a404-ed6cdccfee66\" (UID: \"7b39b8af-d182-45b5-a404-ed6cdccfee66\") "
Nov 03 23:47:05.399337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.399307    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume" pod "7b39b8af-d182-45b5-a404-ed6cdccfee66" (UID: "7b39b8af-d182-45b5-a404-ed6cdccfee66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:05.401595 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.401567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf" pod "7b39b8af-d182-45b5-a404-ed6cdccfee66" (UID: "7b39b8af-d182-45b5-a404-ed6cdccfee66"). InnerVolumeSpecName "kube-api-access-g6jgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:05.445908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.445877    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:47:05.500326 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.500294    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\" (UniqueName: \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\") pod \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") "
Nov 03 23:47:05.500553 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.500483    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\") pod \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\" (UID: \"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62\") "
Nov 03 23:47:05.500909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.500881    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume" pod "fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" (UID: "fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:05.501106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.501085    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:05.501188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.501112    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6jgf\" (UniqueName: \"kubernetes.io/projected/7b39b8af-d182-45b5-a404-ed6cdccfee66-kube-api-access-g6jgf\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:05.501188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.501126    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b39b8af-d182-45b5-a404-ed6cdccfee66-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:05.503160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.503133    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv" pod "fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" (UID: "fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62"). InnerVolumeSpecName "kube-api-access-295zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:05.602942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.602907    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-295zv\" (UniqueName: \"kubernetes.io/projected/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62-kube-api-access-295zv\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:05.666976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.666945    6430 generic.go:356] "Generic (PLEG): container finished" podID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" containerID="9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd" exitCode=0
Nov 03 23:47:05.667149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.666986    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"
Nov 03 23:47:05.667149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.667032    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" event={"ID":"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62","Type":"ContainerDied","Data":"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"}
Nov 03 23:47:05.667149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.667121    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq" event={"ID":"fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62","Type":"ContainerDied","Data":"c0b08102b7521045b2231b3f923981b087cb216fd1227604da7aff6bb38624d8"}
Nov 03 23:47:05.667339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.667149    6430 scope.go:122] "RemoveContainer" containerID="9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"
Nov 03 23:47:05.668635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.668612    6430 generic.go:356] "Generic (PLEG): container finished" podID="7b39b8af-d182-45b5-a404-ed6cdccfee66" containerID="6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1" exitCode=0
Nov 03 23:47:05.668704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.668648    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"
Nov 03 23:47:05.668704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.668659    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" event={"ID":"7b39b8af-d182-45b5-a404-ed6cdccfee66","Type":"ContainerDied","Data":"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"}
Nov 03 23:47:05.668704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.668700    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4" event={"ID":"7b39b8af-d182-45b5-a404-ed6cdccfee66","Type":"ContainerDied","Data":"bab0d3f5878559525ccdae1ecea6835ad28c21e219170e200ceb88103a5204ee"}
Nov 03 23:47:05.678153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.678079    6430 scope.go:122] "RemoveContainer" containerID="9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"
Nov 03 23:47:05.678454 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:05.678424    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd\": container with ID starting with 9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd not found: ID does not exist" containerID="9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"
Nov 03 23:47:05.678538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.678466    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd"} err="failed to get container status \"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd\": rpc error: code = NotFound desc = could not find container \"9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd\": container with ID starting with 9f399e734b06a3b8cef612c6ca595dda13daf45dbc97a218c2298fe1e77019bd not found: ID does not exist"
Nov 03 23:47:05.678538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.678484    6430 scope.go:122] "RemoveContainer" containerID="6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"
Nov 03 23:47:05.687701 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.687680    6430 scope.go:122] "RemoveContainer" containerID="6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"
Nov 03 23:47:05.687994 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:05.687976    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1\": container with ID starting with 6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1 not found: ID does not exist" containerID="6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"
Nov 03 23:47:05.688043 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.688003    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1"} err="failed to get container status \"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1\": rpc error: code = NotFound desc = could not find container \"6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1\": container with ID starting with 6634e084401ef2dc1bfe05d41711bea19f5a0cc89fe8586ddc1c863f59e983a1 not found: ID does not exist"
Nov 03 23:47:05.695767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.695746    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:47:05.701187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.701164    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de63-bb8fbd6f5-r26qq"]
Nov 03 23:47:05.736666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.736631    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:47:05.739576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:05.739549    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de63-67dc8b47db-nd5p4"]
Nov 03 23:47:06.233094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:06.233064    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b39b8af-d182-45b5-a404-ed6cdccfee66" path="/var/lib/kubelet/pods/7b39b8af-d182-45b5-a404-ed6cdccfee66/volumes"
Nov 03 23:47:06.233579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:06.233565    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede29a37-da72-4da3-bc1f-83087e266934" path="/var/lib/kubelet/pods/ede29a37-da72-4da3-bc1f-83087e266934/volumes"
Nov 03 23:47:06.233938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:06.233925    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" path="/var/lib/kubelet/pods/fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62/volumes"
Nov 03 23:47:08.490694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:08.490656    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:47:08.491515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:08.491480    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" podUID="8cad2477-b528-47df-9d1d-68830d853cf8" containerName="istio-proxy" containerID="cri-o://c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9" gracePeriod=30
Nov 03 23:47:10.565610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.565576    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.614118    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.614358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615350    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615461    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615517    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615567    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.615636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615603    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.616119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615729    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.616119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.616119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.615847    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\") pod \"8cad2477-b528-47df-9d1d-68830d853cf8\" (UID: \"8cad2477-b528-47df-9d1d-68830d853cf8\") "
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.617261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.617802    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.618114    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.618624    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.621565    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.621590    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.621606    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.621621    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.635412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.621636    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cad2477-b528-47df-9d1d-68830d853cf8-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.645649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.645614    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:47:10.671736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.671697    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:47:10.685471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.673010    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:10.697347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.697265    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j" pod "8cad2477-b528-47df-9d1d-68830d853cf8" (UID: "8cad2477-b528-47df-9d1d-68830d853cf8"). InnerVolumeSpecName "kube-api-access-w6k4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:10.723158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.723118    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cad2477-b528-47df-9d1d-68830d853cf8-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.723158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.723156    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6k4j\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-kube-api-access-w6k4j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.723375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.723172    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cad2477-b528-47df-9d1d-68830d853cf8-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.723375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.723186    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cad2477-b528-47df-9d1d-68830d853cf8-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:10.857944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.857908    6430 generic.go:356] "Generic (PLEG): container finished" podID="8cad2477-b528-47df-9d1d-68830d853cf8" containerID="c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9" exitCode=0
Nov 03 23:47:10.858129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.857960    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"
Nov 03 23:47:10.858129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.857991    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" event={"ID":"8cad2477-b528-47df-9d1d-68830d853cf8","Type":"ContainerDied","Data":"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"}
Nov 03 23:47:10.858129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.858043    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9" event={"ID":"8cad2477-b528-47df-9d1d-68830d853cf8","Type":"ContainerDied","Data":"bb7bb4553a51ee6d527161f4856864ff1eebf631d0894e5182b381b27e72262c"}
Nov 03 23:47:10.858129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.858068    6430 scope.go:122] "RemoveContainer" containerID="c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"
Nov 03 23:47:10.869062 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.869043    6430 scope.go:122] "RemoveContainer" containerID="c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"
Nov 03 23:47:10.869345 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:10.869325    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9\": container with ID starting with c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9 not found: ID does not exist" containerID="c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"
Nov 03 23:47:10.869426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.869354    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9"} err="failed to get container status \"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9\": rpc error: code = NotFound desc = could not find container \"c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9\": container with ID starting with c32f7e5277d006cb886b905198882eb195b398410ff7d6ec51d98846994630c9 not found: ID does not exist"
Nov 03 23:47:10.888733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.888703    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:47:10.893445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:10.893421    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-32de63-openshift-default-5f88dd7497-4c7p9"]
Nov 03 23:47:12.232159 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:12.232124    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cad2477-b528-47df-9d1d-68830d853cf8" path="/var/lib/kubelet/pods/8cad2477-b528-47df-9d1d-68830d853cf8/volumes"
Nov 03 23:47:14.013733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.013698    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7308-444df7"]
Nov 03 23:47:14.014333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014319    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8cad2477-b528-47df-9d1d-68830d853cf8" containerName="istio-proxy"
Nov 03 23:47:14.014399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014334    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cad2477-b528-47df-9d1d-68830d853cf8" containerName="istio-proxy"
Nov 03 23:47:14.014399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014346    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:14.014399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014350    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:14.014399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014378    6430 cpu_manager.go:414] "containerMap: removing container" podUID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" containerName="payload-pre-processing"
Nov 03 23:47:14.014399 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014383    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" containerName="payload-pre-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014410    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7b39b8af-d182-45b5-a404-ed6cdccfee66" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014416    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b39b8af-d182-45b5-a404-ed6cdccfee66" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014426    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014431    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014438    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014442    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014448    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" containerName="payload-pre-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014452    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" containerName="payload-pre-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014472    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c936f359-9af4-483e-bb97-bd4b0c687a0a" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014478    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c936f359-9af4-483e-bb97-bd4b0c687a0a" containerName="payload-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014487    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014491    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014500    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" containerName="payload-pre-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014505    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" containerName="payload-pre-processing"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014512    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ede29a37-da72-4da3-bc1f-83087e266934" containerName="maas-api"
Nov 03 23:47:14.014586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014516    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede29a37-da72-4da3-bc1f-83087e266934" containerName="maas-api"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014600    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c936f359-9af4-483e-bb97-bd4b0c687a0a" containerName="payload-processing"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014610    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014616    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48f41c3-de2d-4d7e-af93-d8da0d06828f" containerName="payload-pre-processing"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014621    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede29a37-da72-4da3-bc1f-83087e266934" containerName="maas-api"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014626    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cad2477-b528-47df-9d1d-68830d853cf8" containerName="istio-proxy"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014632    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014641    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="124a50ab-fb9b-49d0-aa06-4a6986c3d908" containerName="payload-pre-processing"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014647    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b39b8af-d182-45b5-a404-ed6cdccfee66" containerName="payload-processing"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014653    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec7fad9-4c78-4eae-8be3-cb2225fb56cd" containerName="payload-processing"
Nov 03 23:47:14.015048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.014658    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd18f2c3-ec81-47bb-9d9c-d6fa0678cf62" containerName="payload-pre-processing"
Nov 03 23:47:14.019465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.019447    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:14.024301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.024272    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hxqlx\""
Nov 03 23:47:14.025510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.025483    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7308-444df7"]
Nov 03 23:47:14.088190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.088145    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbph2\" (UniqueName: \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\") pod \"test-curl-7308-444df7\" (UID: \"534fc56b-1d59-4f0b-ac83-a599e9f3a420\") " pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:14.189326 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.189289    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbph2\" (UniqueName: \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\") pod \"test-curl-7308-444df7\" (UID: \"534fc56b-1d59-4f0b-ac83-a599e9f3a420\") " pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:14.209316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.209278    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbph2\" (UniqueName: \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\") pod \"test-curl-7308-444df7\" (UID: \"534fc56b-1d59-4f0b-ac83-a599e9f3a420\") " pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:14.333873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.333838    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:14.616032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:14.616003    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7308-444df7"]
Nov 03 23:47:14.616778 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:14.616744    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod534fc56b_1d59_4f0b_ac83_a599e9f3a420.slice/crio-cd8fe800c21b0db20be1830f2018e124ef1965a33c24545dba7406bb8393ca26 WatchSource:0}: Error finding container cd8fe800c21b0db20be1830f2018e124ef1965a33c24545dba7406bb8393ca26: Status 404 returned error can't find the container with id cd8fe800c21b0db20be1830f2018e124ef1965a33c24545dba7406bb8393ca26
Nov 03 23:47:15.014459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:15.006816    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7308-444df7" event={"ID":"534fc56b-1d59-4f0b-ac83-a599e9f3a420","Type":"ContainerStarted","Data":"cd8fe800c21b0db20be1830f2018e124ef1965a33c24545dba7406bb8393ca26"}
Nov 03 23:47:16.015262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:16.015168    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7308-444df7" event={"ID":"534fc56b-1d59-4f0b-ac83-a599e9f3a420","Type":"ContainerStarted","Data":"9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4"}
Nov 03 23:47:16.036440 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:16.036357    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7308-444df7" podStartSLOduration=1.976412359 podStartE2EDuration="3.036342318s" totalImagesPullingTime="1.059929959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:16.035499782 +0000 UTC m=+1051.926561737" watchObservedRunningTime="2026-11-03 23:47:16.036342318 +0000 UTC m=+1051.927404253"
Nov 03 23:47:16.756743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:16.756702    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7308-444df7"]
Nov 03 23:47:16.769175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:16.769074    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7308-444df7"]
Nov 03 23:47:18.038183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:18.038125    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7308-444df7" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" containerName="test-curl-7308-444df7" containerID="cri-o://9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4" gracePeriod=2
Nov 03 23:47:18.041645 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:18.041613    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7308-444df7\" is forbidden: User \"system:node:ip-10-0-143-134.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-east-2.compute.internal' and this object" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:20.248401 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:20.248365    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:20.294908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:20.294830    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\" (UniqueName: \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\") pod \"534fc56b-1d59-4f0b-ac83-a599e9f3a420\" (UID: \"534fc56b-1d59-4f0b-ac83-a599e9f3a420\") "
Nov 03 23:47:20.297705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:20.297659    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2" pod "534fc56b-1d59-4f0b-ac83-a599e9f3a420" (UID: "534fc56b-1d59-4f0b-ac83-a599e9f3a420"). InnerVolumeSpecName "kube-api-access-wbph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:20.406579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:20.404014    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbph2\" (UniqueName: \"kubernetes.io/projected/534fc56b-1d59-4f0b-ac83-a599e9f3a420-kube-api-access-wbph2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:21.059472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:21.059428    6430 generic.go:356] "Generic (PLEG): container finished" podID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" containerID="9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4" exitCode=137
Nov 03 23:47:21.059670 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:21.059510    6430 scope.go:122] "RemoveContainer" containerID="9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4"
Nov 03 23:47:21.059670 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:21.059521    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:21.069185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:21.069163    6430 scope.go:122] "RemoveContainer" containerID="9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4"
Nov 03 23:47:21.069521 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:21.069497    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4\": container with ID starting with 9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4 not found: ID does not exist" containerID="9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4"
Nov 03 23:47:21.069650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:21.069531    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4"} err="failed to get container status \"9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4\": rpc error: code = NotFound desc = could not find container \"9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4\": container with ID starting with 9d6574e2184d5b0174ef7f26f3368319eeff0d750ab75b27df561975763b6df4 not found: ID does not exist"
Nov 03 23:47:21.071425 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:21.071375    6430 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7308-444df7\" is forbidden: User \"system:node:ip-10-0-143-134.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-east-2.compute.internal' and this object" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" pod="opendatahub/test-curl-7308-444df7"
Nov 03 23:47:22.239921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:22.239877    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" path="/var/lib/kubelet/pods/534fc56b-1d59-4f0b-ac83-a599e9f3a420/volumes"
Nov 03 23:47:27.555206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:27.555164    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:47:28.421324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.421280    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:47:28.461121 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.461084    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:28.461342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.461324    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:28.461618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.461602    6430 cpu_manager.go:414] "containerMap: removing container" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" containerName="test-curl-7308-444df7"
Nov 03 23:47:28.461725 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.461710    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" containerName="test-curl-7308-444df7"
Nov 03 23:47:28.462152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.462131    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9526b9-75de-4674-9b33-1af6ee23abfa" containerName="istio-proxy"
Nov 03 23:47:28.463542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.463517    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3f2edb-bc84-41fe-bf85-af7b44eb6a58" containerName="istio-proxy"
Nov 03 23:47:28.463648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.463557    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="534fc56b-1d59-4f0b-ac83-a599e9f3a420" containerName="test-curl-7308-444df7"
Nov 03 23:47:28.476594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.476554    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:47:28.476714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.476684    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.490804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.490780    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9bf431-openshift-default-dockercfg-z7jk6\""
Nov 03 23:47:28.574545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.538879    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.575362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.575327    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.600560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.600516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vf6s\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.600852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.600824    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.601031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.601010    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.601222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.601204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.601372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.601356    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.601739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.601717    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.601870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.601856    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728510    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728557    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728607    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728665    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728728    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vf6s\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728770    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728880    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.728941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.730892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.730668    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.731547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.730992    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.731692 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.731630    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.744418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.732658    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.744418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.733205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.749413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.746614    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.778409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.774405    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.778409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.776073    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vf6s\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:28.795081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:28.795050    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\") pod \"e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") " pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:29.094736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.094701    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:29.387222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.387191    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:47:29.387885 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:29.387860    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960332d4_0549_4a97_9573_c8204aec94ab.slice/crio-fdfe48857d5f2bc7fa5c4eed2c333e749cf0c8be7fd56c44277a0280be47b7dc WatchSource:0}: Error finding container fdfe48857d5f2bc7fa5c4eed2c333e749cf0c8be7fd56c44277a0280be47b7dc: Status 404 returned error can't find the container with id fdfe48857d5f2bc7fa5c4eed2c333e749cf0c8be7fd56c44277a0280be47b7dc
Nov 03 23:47:29.390292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.390261    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:29.390364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.390324    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:29.390418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.390372    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:29.684850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:29.684768    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:47:30.184207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:30.184163    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerStarted","Data":"672c78192b631c0f953db5db44b29292dfd623135a898bf952803cba000cb13a"}
Nov 03 23:47:30.184408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:30.184226    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerStarted","Data":"fdfe48857d5f2bc7fa5c4eed2c333e749cf0c8be7fd56c44277a0280be47b7dc"}
Nov 03 23:47:30.217912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:30.217856    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podStartSLOduration=2.217841486 podStartE2EDuration="2.217841486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:30.214760622 +0000 UTC m=+1066.105822557" watchObservedRunningTime="2026-11-03 23:47:30.217841486 +0000 UTC m=+1066.108903421"
Nov 03 23:47:31.095445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:31.095410    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:31.101477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:31.101451    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:31.191176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:31.191144    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:31.192109 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:31.192090    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:47:32.556441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:32.556373    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" podUID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" containerName="payload-pre-processing" containerID="cri-o://ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928" gracePeriod=25
Nov 03 23:47:33.064736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.064712    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:47:33.145366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.145328    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\") pod \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") "
Nov 03 23:47:33.145613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.145376    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\") pod \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\" (UID: \"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2\") "
Nov 03 23:47:33.145846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.145820    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume" pod "999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" (UID: "999d7568-59ab-4ef6-b3d1-98dc2d81c5f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:47:33.145992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.145978    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:33.147579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.147554    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd" pod "999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" (UID: "999d7568-59ab-4ef6-b3d1-98dc2d81c5f2"). InnerVolumeSpecName "kube-api-access-z24pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:47:33.207339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.207307    6430 generic.go:356] "Generic (PLEG): container finished" podID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" containerID="ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928" exitCode=0
Nov 03 23:47:33.207527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.207362    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"
Nov 03 23:47:33.207527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.207431    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" event={"ID":"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2","Type":"ContainerDied","Data":"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"}
Nov 03 23:47:33.207527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.207499    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5" event={"ID":"999d7568-59ab-4ef6-b3d1-98dc2d81c5f2","Type":"ContainerDied","Data":"fd8029f18e7b0a03580a6bc283175a57d9a24cb264a4777756cb94e72d350158"}
Nov 03 23:47:33.207684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.207527    6430 scope.go:122] "RemoveContainer" containerID="ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"
Nov 03 23:47:33.218477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.218456    6430 scope.go:122] "RemoveContainer" containerID="ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"
Nov 03 23:47:33.218753 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:33.218732    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928\": container with ID starting with ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928 not found: ID does not exist" containerID="ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"
Nov 03 23:47:33.218802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.218762    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928"} err="failed to get container status \"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928\": rpc error: code = NotFound desc = could not find container \"ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928\": container with ID starting with ac434de15e8889ce47db5c2fd8f1de23aa764eda2cbe30836f373bfc7b7ba928 not found: ID does not exist"
Nov 03 23:47:33.247757 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.247415    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2-kube-api-access-z24pd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:47:33.250817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.250786    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:47:33.253412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.253375    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-j9dt5"]
Nov 03 23:47:33.897449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.897371    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:47:33.897993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.897978    6430 cpu_manager.go:414] "containerMap: removing container" podUID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" containerName="payload-pre-processing"
Nov 03 23:47:33.898042 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.897994    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" containerName="payload-pre-processing"
Nov 03 23:47:33.898113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.898104    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" containerName="payload-pre-processing"
Nov 03 23:47:33.903278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.903258    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:33.917698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.917666    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9bf431\""
Nov 03 23:47:33.918473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.918447    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:47:33.921353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.921329    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:47:33.925781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.925759    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:33.933887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.933864    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9bf431-dockercfg-l99fc\""
Nov 03 23:47:33.946554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.946527    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:47:33.965100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.965066    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:33.965241 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.965210    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6s2\" (UniqueName: \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:33.965361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.965347    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:33.965468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:33.965376    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdth\" (UniqueName: \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.085756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.066860    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:34.100406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.097837    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdth\" (UniqueName: \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.100406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.098115    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.100406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.098210    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6s2\" (UniqueName: \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:34.100406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.073323    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:34.100406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.099589    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.128412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.124327    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdth\" (UniqueName: \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\") pod \"payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.128667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.126236    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6s2\" (UniqueName: \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\") pod \"payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:34.234312 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.234223    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:34.259353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.256530    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:34.335684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.335648    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999d7568-59ab-4ef6-b3d1-98dc2d81c5f2" path="/var/lib/kubelet/pods/999d7568-59ab-4ef6-b3d1-98dc2d81c5f2/volumes"
Nov 03 23:47:34.641810 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.641780    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:47:34.642357 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:34.642325    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7b4d71_a7ac_4796_8b9f_79db25db1c06.slice/crio-a1b5504cd347ca790ab67c292761509b6ac430e003b1bca1371bb52e274318d2 WatchSource:0}: Error finding container a1b5504cd347ca790ab67c292761509b6ac430e003b1bca1371bb52e274318d2: Status 404 returned error can't find the container with id a1b5504cd347ca790ab67c292761509b6ac430e003b1bca1371bb52e274318d2
Nov 03 23:47:34.643132 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:34.643095    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771b6c87_0b69_4e64_9736_74fe3be72437.slice/crio-653dfbce6a9e655f96bda14fd51acd7c3d30c25740d95ca400848a4657970af8 WatchSource:0}: Error finding container 653dfbce6a9e655f96bda14fd51acd7c3d30c25740d95ca400848a4657970af8: Status 404 returned error can't find the container with id 653dfbce6a9e655f96bda14fd51acd7c3d30c25740d95ca400848a4657970af8
Nov 03 23:47:34.644931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:34.644904    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:47:34.836147 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:47:34.836107    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:47:35.341889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.341848    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" event={"ID":"2d7b4d71-a7ac-4796-8b9f-79db25db1c06","Type":"ContainerStarted","Data":"db0d7b1a8e97ecd918a7d651fc1d73ed0ce1d8226501eb1c2bd5735150f2de30"}
Nov 03 23:47:35.342328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.341907    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" event={"ID":"2d7b4d71-a7ac-4796-8b9f-79db25db1c06","Type":"ContainerStarted","Data":"a1b5504cd347ca790ab67c292761509b6ac430e003b1bca1371bb52e274318d2"}
Nov 03 23:47:35.342328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.341935    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:35.343555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.343522    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" event={"ID":"771b6c87-0b69-4e64-9736-74fe3be72437","Type":"ContainerStarted","Data":"f9eb98bbe314c5d47e047719e7e71b8477cad8a1d6e2ed6af235ebaf5541b23b"}
Nov 03 23:47:35.343694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.343569    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" event={"ID":"771b6c87-0b69-4e64-9736-74fe3be72437","Type":"ContainerStarted","Data":"653dfbce6a9e655f96bda14fd51acd7c3d30c25740d95ca400848a4657970af8"}
Nov 03 23:47:35.343694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.343589    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:35.363631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.363574    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" podStartSLOduration=2.363562346 podStartE2EDuration="2.363562346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:35.362176228 +0000 UTC m=+1071.253238163" watchObservedRunningTime="2026-11-03 23:47:35.363562346 +0000 UTC m=+1071.254624280"
Nov 03 23:47:35.388287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:35.388232    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" podStartSLOduration=2.388216711 podStartE2EDuration="2.388216711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:35.386721177 +0000 UTC m=+1071.277783113" watchObservedRunningTime="2026-11-03 23:47:35.388216711 +0000 UTC m=+1071.279278646"
Nov 03 23:47:36.642887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.641433    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:47:36.646285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.646259    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.659928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.659894    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:47:36.660078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.660032    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e4892d-openshift-default-dockercfg-k77dh\""
Nov 03 23:47:36.753838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.753803    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.753838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.753845    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpq7\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.753869    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.753923    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.753948    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.754172    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.754206    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.754240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.754307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.754287    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.859495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.859450    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.860205 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.860177    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.860419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.860382    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.860565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.860549    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.860827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.860806    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.877460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.877420    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpq7\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.877734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.877715    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.877851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.877837    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.877963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.877950    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.879967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.861730    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.880100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.860122    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.880301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.862373    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.880643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.880615    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.881194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.881168    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.886227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.886159    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.910956 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.886655    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.930538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.912914    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.930768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.923640    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpq7\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\") pod \"e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") " pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:36.970452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:36.966662    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:37.395952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.395916    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:47:37.403502 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:37.403459    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c125e32_7920_4c23_aa7d_f9bc57baa92a.slice/crio-a0251fad7fc18965a0c4bb842e45f8670793539aaeb8636f3d5644940c51bd64 WatchSource:0}: Error finding container a0251fad7fc18965a0c4bb842e45f8670793539aaeb8636f3d5644940c51bd64: Status 404 returned error can't find the container with id a0251fad7fc18965a0c4bb842e45f8670793539aaeb8636f3d5644940c51bd64
Nov 03 23:47:37.406571 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.406530    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:37.406687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.406616    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:37.406687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.406660    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:37.425490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.425457    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:47:37.435516 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.435488    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:47:37.435644 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.435626    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.441607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.441582    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9bf431-serving-cert\""
Nov 03 23:47:37.499357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.499329    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.499540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.499366    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.499540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.499511    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggsw\" (UniqueName: \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.499743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.499722    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.601098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.601055    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.601304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.601191    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.601304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.601264    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggsw\" (UniqueName: \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.601491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.601470    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.602493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.602460    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.608006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.607928    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.609030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.609008    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.637980 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.632178    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggsw\" (UniqueName: \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\") pod \"maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:37.773075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:37.772973    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:38.076140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.076109    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:47:38.077791 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:38.077752    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a11c94_9578_4e43_b02a_7feee8281935.slice/crio-26ab7e8d34f8fc54780347a897b64a4eace9eed4c733c9462b4605970c3248bd WatchSource:0}: Error finding container 26ab7e8d34f8fc54780347a897b64a4eace9eed4c733c9462b4605970c3248bd: Status 404 returned error can't find the container with id 26ab7e8d34f8fc54780347a897b64a4eace9eed4c733c9462b4605970c3248bd
Nov 03 23:47:38.368225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.368122    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" event={"ID":"02a11c94-9578-4e43-b02a-7feee8281935","Type":"ContainerStarted","Data":"26ab7e8d34f8fc54780347a897b64a4eace9eed4c733c9462b4605970c3248bd"}
Nov 03 23:47:38.371448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.370166    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerStarted","Data":"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"}
Nov 03 23:47:38.371448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.370222    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerStarted","Data":"a0251fad7fc18965a0c4bb842e45f8670793539aaeb8636f3d5644940c51bd64"}
Nov 03 23:47:38.397624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.397575    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podStartSLOduration=2.397561723 podStartE2EDuration="2.397561723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:38.396939469 +0000 UTC m=+1074.288010719" watchObservedRunningTime="2026-11-03 23:47:38.397561723 +0000 UTC m=+1074.288623657"
Nov 03 23:47:38.967965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.967875    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:38.973140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:38.973114    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:39.394329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:39.383284    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" event={"ID":"02a11c94-9578-4e43-b02a-7feee8281935","Type":"ContainerStarted","Data":"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"}
Nov 03 23:47:39.394329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:39.384989    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:39.394329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:39.385073    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:39.394329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:39.386350    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:47:39.419410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:39.418424    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" podStartSLOduration=1.8121232840000001 podStartE2EDuration="2.418363534s" totalImagesPullingTime="606.24025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:39.408994233 +0000 UTC m=+1075.300056215" watchObservedRunningTime="2026-11-03 23:47:39.418363534 +0000 UTC m=+1075.309425469"
Nov 03 23:47:41.999060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:41.999023    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:47:42.003720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.003693    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.012831 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.012806    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:47:42.016649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.016628    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e4892d\""
Nov 03 23:47:42.018406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.018375    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:47:42.018522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.018485    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.029240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.028529    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e4892d-dockercfg-pnqsx\""
Nov 03 23:47:42.045518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.045488    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:47:42.159425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.156345    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whh4w\" (UniqueName: \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.159425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.156473    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.159425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.156556    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.159425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.156846    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj48t\" (UniqueName: \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.298459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.295461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whh4w\" (UniqueName: \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.298459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.295571    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.298459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.295671    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.298459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.296009    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj48t\" (UniqueName: \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.375413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.362566    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.402416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.388211    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.435433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.427712    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj48t\" (UniqueName: \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\") pod \"payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.452422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.447212    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whh4w\" (UniqueName: \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\") pod \"payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.638380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.637696    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:42.638380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.638228    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:42.933534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.933447    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:47:42.934160 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:42.934133    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd01a30_7a50_4647_b306_8b014abe33e2.slice/crio-602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3 WatchSource:0}: Error finding container 602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3: Status 404 returned error can't find the container with id 602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3
Nov 03 23:47:42.936710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:42.936686    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:47:42.937478 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:42.937449    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41178d30_3421_4aeb_8b53_0cdd278604a0.slice/crio-02dc083e4bbfa1704a540f00b35509181960dbccbb9922ef84fc928c7581485c WatchSource:0}: Error finding container 02dc083e4bbfa1704a540f00b35509181960dbccbb9922ef84fc928c7581485c: Status 404 returned error can't find the container with id 02dc083e4bbfa1704a540f00b35509181960dbccbb9922ef84fc928c7581485c
Nov 03 23:47:43.464463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.464379    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" event={"ID":"41178d30-3421-4aeb-8b53-0cdd278604a0","Type":"ContainerStarted","Data":"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"}
Nov 03 23:47:43.464925 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.464595    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:43.464925 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.464641    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" event={"ID":"41178d30-3421-4aeb-8b53-0cdd278604a0","Type":"ContainerStarted","Data":"02dc083e4bbfa1704a540f00b35509181960dbccbb9922ef84fc928c7581485c"}
Nov 03 23:47:43.466266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.466235    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" event={"ID":"9dd01a30-7a50-4647-b306-8b014abe33e2","Type":"ContainerStarted","Data":"6a388b90973cb4c686654cfb14af6c9856b3ed7f39f97594d9f0ebedec499bcd"}
Nov 03 23:47:43.466416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.466277    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" event={"ID":"9dd01a30-7a50-4647-b306-8b014abe33e2","Type":"ContainerStarted","Data":"602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3"}
Nov 03 23:47:43.466416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.466323    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:43.487092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.487042    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" podStartSLOduration=2.487028067 podStartE2EDuration="2.487028067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:43.486569462 +0000 UTC m=+1079.377631410" watchObservedRunningTime="2026-11-03 23:47:43.487028067 +0000 UTC m=+1079.378090002"
Nov 03 23:47:43.519548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:43.519481    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" podStartSLOduration=2.519460847 podStartE2EDuration="2.519460847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:43.515627845 +0000 UTC m=+1079.406689781" watchObservedRunningTime="2026-11-03 23:47:43.519460847 +0000 UTC m=+1079.410522786"
Nov 03 23:47:45.400347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:45.400318    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:47:46.354160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:46.354129    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:47:46.354327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:46.354272    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:47:47.045107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.039172    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:47:47.118659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.118621    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:47:47.159709 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.134563    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.177302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.177265    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d41841-openshift-default-dockercfg-8xddc\""
Nov 03 23:47:47.277949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.277537    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlhj5\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.278352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.278327    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.278681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.278652    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.278872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.278855    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.279051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.279032    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.279168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.279152    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.279354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.279334    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.279598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.279576    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.279726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.279709    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.384565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.384520    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.384999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.384973    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385192 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385176    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385318    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385441    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385600    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385765    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.385898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.385883    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.386095 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.386078    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlhj5\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.387063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.387037    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.396918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.396880    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.407408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.402302    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.407408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.402356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.407408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.402768    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.449418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.440965    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.449418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.442618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.519860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.494441    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.623016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.618265    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlhj5\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\") pod \"e2e-api-a-d41841-openshift-default-d6d57f569-xlmft\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") " pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:47.624582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:47.624550    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:48.147002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.146975    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:47:48.148813 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:48.148787    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32383bd_1f7a_4665_a069_732ad7b74293.slice/crio-60ef6fedd3198ecc542fc51946a6a075c2fbb22039e3167c258ec1ecd3b5f8c5 WatchSource:0}: Error finding container 60ef6fedd3198ecc542fc51946a6a075c2fbb22039e3167c258ec1ecd3b5f8c5: Status 404 returned error can't find the container with id 60ef6fedd3198ecc542fc51946a6a075c2fbb22039e3167c258ec1ecd3b5f8c5
Nov 03 23:47:48.151457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.151421    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:48.151527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.151497    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:48.151564 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.151529    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:47:48.768416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.764463    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerStarted","Data":"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"}
Nov 03 23:47:48.768416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.764537    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerStarted","Data":"60ef6fedd3198ecc542fc51946a6a075c2fbb22039e3167c258ec1ecd3b5f8c5"}
Nov 03 23:47:48.825416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:48.823603    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podStartSLOduration=2.823580448 podStartE2EDuration="2.823580448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:48.805773462 +0000 UTC m=+1084.696835397" watchObservedRunningTime="2026-11-03 23:47:48.823580448 +0000 UTC m=+1084.714642384"
Nov 03 23:47:49.625234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:49.625197    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:49.631280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:49.631251    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:49.806416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:49.802439    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:49.806416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:49.802552    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:47:52.558230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.558191    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:47:52.562751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.562730    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.568066 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.568042    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d41841\""
Nov 03 23:47:52.577663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.577640    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:47:52.626247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.626212    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d41841-dockercfg-4d8k7\""
Nov 03 23:47:52.737575 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.737533    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.750047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.750000    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvc4\" (UniqueName: \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.857821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.857723    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.859353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.859332    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvc4\" (UniqueName: \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.860270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.859237    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:52.881204 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:52.881167    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvc4\" (UniqueName: \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\") pod \"payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:53.183124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:53.182719    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:53.507064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:53.507035    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:47:53.508038 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:53.507974    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ea1ffd_9eec_47fe_ae6e_8a1ea30a92b2.slice/crio-5b1195354d04d3fddd3f1b4a265f19bdf69281cc9f436d44f9ff0a6c9f109dbb WatchSource:0}: Error finding container 5b1195354d04d3fddd3f1b4a265f19bdf69281cc9f436d44f9ff0a6c9f109dbb: Status 404 returned error can't find the container with id 5b1195354d04d3fddd3f1b4a265f19bdf69281cc9f436d44f9ff0a6c9f109dbb
Nov 03 23:47:53.977410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:53.976200    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" event={"ID":"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2","Type":"ContainerStarted","Data":"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"}
Nov 03 23:47:53.977410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:53.976320    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" event={"ID":"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2","Type":"ContainerStarted","Data":"5b1195354d04d3fddd3f1b4a265f19bdf69281cc9f436d44f9ff0a6c9f109dbb"}
Nov 03 23:47:53.977972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:53.977525    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:47:54.475631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:54.475592    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:47:54.475812 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:54.475671    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:47:54.498317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:54.498274    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" podStartSLOduration=2.498260051 podStartE2EDuration="2.498260051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:54.023359072 +0000 UTC m=+1089.914421007" watchObservedRunningTime="2026-11-03 23:47:54.498260051 +0000 UTC m=+1090.389322036"
Nov 03 23:47:58.300078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.300038    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:47:58.305081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.305052    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.344244 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.344211    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d41841-serving-cert\""
Nov 03 23:47:58.350744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.350507    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:47:58.460415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.455875    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.460415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.456055    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.460415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.456231    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hv7\" (UniqueName: \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.460415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.456287    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.563473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.562472    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.566422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.563871    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.566422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.564018    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hv7\" (UniqueName: \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.566422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.564056    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.580424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.580366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.597748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.597711    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hv7\" (UniqueName: \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.611441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.611406    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.619431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.619379    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\") pod \"maas-api-e2e-api-a-d41841-5776877d8b-qtsb6\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:58.660132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:58.660096    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:47:59.174112 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:47:59.174048    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d320f12_1741_4fe2_9064_c883ab6a12e5.slice/crio-9af356daaf360436a5e862f13d077da20964918407bebbd00b3f21c8651c1213 WatchSource:0}: Error finding container 9af356daaf360436a5e862f13d077da20964918407bebbd00b3f21c8651c1213: Status 404 returned error can't find the container with id 9af356daaf360436a5e862f13d077da20964918407bebbd00b3f21c8651c1213
Nov 03 23:47:59.193583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:47:59.193553    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:48:00.192743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.192706    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" event={"ID":"1d320f12-1741-4fe2-9064-c883ab6a12e5","Type":"ContainerStarted","Data":"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"}
Nov 03 23:48:00.193259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.192785    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:48:00.193259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.192811    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" event={"ID":"1d320f12-1741-4fe2-9064-c883ab6a12e5","Type":"ContainerStarted","Data":"9af356daaf360436a5e862f13d077da20964918407bebbd00b3f21c8651c1213"}
Nov 03 23:48:00.219787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.219732    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" podStartSLOduration=1.652250978 podStartE2EDuration="2.219719326s" totalImagesPullingTime="567.468348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:00.216149029 +0000 UTC m=+1096.107210964" watchObservedRunningTime="2026-11-03 23:48:00.219719326 +0000 UTC m=+1096.110781262"
Nov 03 23:48:00.823171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.823134    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:48:00.845236 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:00.845185    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:48:02.234540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:02.234508    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:02.234937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:02.234561    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:04.885308 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:48:04.885263    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:48:05.249257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.249165    6430 generic.go:356] "Generic (PLEG): container finished" podID="960332d4-0549-4a97-9573-c8204aec94ab" containerID="672c78192b631c0f953db5db44b29292dfd623135a898bf952803cba000cb13a" exitCode=0
Nov 03 23:48:05.249418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.249242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerDied","Data":"672c78192b631c0f953db5db44b29292dfd623135a898bf952803cba000cb13a"}
Nov 03 23:48:05.249851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.249835    6430 scope.go:122] "RemoveContainer" containerID="672c78192b631c0f953db5db44b29292dfd623135a898bf952803cba000cb13a"
Nov 03 23:48:05.251681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.251653    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:05.251751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.251724    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:05.251786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.251758    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:05.823945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.823877    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" podUID="41178d30-3421-4aeb-8b53-0cdd278604a0" containerName="payload-processing" containerID="cri-o://4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c" gracePeriod=25
Nov 03 23:48:05.846632 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:05.846566    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" podUID="9dd01a30-7a50-4647-b306-8b014abe33e2" containerName="payload-pre-processing" containerID="cri-o://6a388b90973cb4c686654cfb14af6c9856b3ed7f39f97594d9f0ebedec499bcd" gracePeriod=25
Nov 03 23:48:06.040456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.039060    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:48:06.199220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.199193    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:48:06.271044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.270505    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:48:06.273147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.273126    6430 generic.go:356] "Generic (PLEG): container finished" podID="41178d30-3421-4aeb-8b53-0cdd278604a0" containerID="4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c" exitCode=0
Nov 03 23:48:06.273335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.273221    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"
Nov 03 23:48:06.273457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.273238    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" event={"ID":"41178d30-3421-4aeb-8b53-0cdd278604a0","Type":"ContainerDied","Data":"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"}
Nov 03 23:48:06.273457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.273446    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj" event={"ID":"41178d30-3421-4aeb-8b53-0cdd278604a0","Type":"ContainerDied","Data":"02dc083e4bbfa1704a540f00b35509181960dbccbb9922ef84fc928c7581485c"}
Nov 03 23:48:06.273600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.273485    6430 scope.go:122] "RemoveContainer" containerID="4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"
Nov 03 23:48:06.277500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.277451    6430 generic.go:356] "Generic (PLEG): container finished" podID="9dd01a30-7a50-4647-b306-8b014abe33e2" containerID="6a388b90973cb4c686654cfb14af6c9856b3ed7f39f97594d9f0ebedec499bcd" exitCode=0
Nov 03 23:48:06.277612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.277488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" event={"ID":"9dd01a30-7a50-4647-b306-8b014abe33e2","Type":"ContainerDied","Data":"6a388b90973cb4c686654cfb14af6c9856b3ed7f39f97594d9f0ebedec499bcd"}
Nov 03 23:48:06.277612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.277558    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc" event={"ID":"9dd01a30-7a50-4647-b306-8b014abe33e2","Type":"ContainerDied","Data":"602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3"}
Nov 03 23:48:06.277612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.277584    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602672efa3967a914e9096d2ce2d1d650e9ed1ea84fb3187f05004564c9f0cd3"
Nov 03 23:48:06.280802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.279884    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerStarted","Data":"0f01b975580cfd4e20699fb9828f67302343ce6855dcf33eb96304d9922d0dad"}
Nov 03 23:48:06.286862 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.286837    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:48:06.287176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.287159    6430 scope.go:122] "RemoveContainer" containerID="4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"
Nov 03 23:48:06.287535 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:48:06.287510    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c\": container with ID starting with 4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c not found: ID does not exist" containerID="4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"
Nov 03 23:48:06.287618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.287547    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c"} err="failed to get container status \"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c\": rpc error: code = NotFound desc = could not find container \"4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c\": container with ID starting with 4fe51f850ed6a08bbad0f7a40ec19a6266c9a7b69ee60d6f4bdbc6bf8476466c not found: ID does not exist"
Nov 03 23:48:06.305137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.303896    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\") pod \"41178d30-3421-4aeb-8b53-0cdd278604a0\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") "
Nov 03 23:48:06.305137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.304037    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\" (UniqueName: \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\") pod \"41178d30-3421-4aeb-8b53-0cdd278604a0\" (UID: \"41178d30-3421-4aeb-8b53-0cdd278604a0\") "
Nov 03 23:48:06.305714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.305670    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume" pod "41178d30-3421-4aeb-8b53-0cdd278604a0" (UID: "41178d30-3421-4aeb-8b53-0cdd278604a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:48:06.337332 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.337221    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t" pod "41178d30-3421-4aeb-8b53-0cdd278604a0" (UID: "41178d30-3421-4aeb-8b53-0cdd278604a0"). InnerVolumeSpecName "kube-api-access-gj48t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:48:06.406338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.406307    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\" (UniqueName: \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\") pod \"9dd01a30-7a50-4647-b306-8b014abe33e2\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") "
Nov 03 23:48:06.406585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.406466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\") pod \"9dd01a30-7a50-4647-b306-8b014abe33e2\" (UID: \"9dd01a30-7a50-4647-b306-8b014abe33e2\") "
Nov 03 23:48:06.407700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.407668    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41178d30-3421-4aeb-8b53-0cdd278604a0-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:06.407700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.407700    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj48t\" (UniqueName: \"kubernetes.io/projected/41178d30-3421-4aeb-8b53-0cdd278604a0-kube-api-access-gj48t\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:06.409630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.409604    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume" pod "9dd01a30-7a50-4647-b306-8b014abe33e2" (UID: "9dd01a30-7a50-4647-b306-8b014abe33e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:48:06.414592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.414564    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w" pod "9dd01a30-7a50-4647-b306-8b014abe33e2" (UID: "9dd01a30-7a50-4647-b306-8b014abe33e2"). InnerVolumeSpecName "kube-api-access-whh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:48:06.532785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.532741    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whh4w\" (UniqueName: \"kubernetes.io/projected/9dd01a30-7a50-4647-b306-8b014abe33e2-kube-api-access-whh4w\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:06.532785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.532779    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dd01a30-7a50-4647-b306-8b014abe33e2-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:06.685413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.684838    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:48:06.714413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:06.713774    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4892d-5f88644f88-mvnhj"]
Nov 03 23:48:07.095181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.095150    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:48:07.095600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.095442    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:07.095600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.095486    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:07.288910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.288877    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"
Nov 03 23:48:07.334212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.334174    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:48:07.337211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.337187    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4892d-6bc8c64687-qd5tc"]
Nov 03 23:48:07.759420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.753449    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:48:07.790412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:07.784434    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:48:08.099430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:08.096877    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:08.099430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:08.097024    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:08.417366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:08.417281    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41178d30-3421-4aeb-8b53-0cdd278604a0" path="/var/lib/kubelet/pods/41178d30-3421-4aeb-8b53-0cdd278604a0/volumes"
Nov 03 23:48:08.418511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:08.418485    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd01a30-7a50-4647-b306-8b014abe33e2" path="/var/lib/kubelet/pods/9dd01a30-7a50-4647-b306-8b014abe33e2/volumes"
Nov 03 23:48:08.661542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:08.661501    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:48:09.095074 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:09.095036    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:09.095285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:09.095104    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:10.168919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:10.168882    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:10.179598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:10.179547    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:11.154414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.139556    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:11.154414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.139645    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:11.181417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.178899    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:11.214416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.206142    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:48:11.289573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.289521    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:11.346454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.343586    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:11.674700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.674644    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:11.966038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:11.965912    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.114843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.114790    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:48:12.116314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.116281    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:12.116541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.116514    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:12.120127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.120095    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.179224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.179182    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:48:12.179979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.179957    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:48:12.222683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.221273    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.222683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.221688    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:48:12.317077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.317027    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.350708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.350655    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.427413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.413594    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:12.427413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.413666    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.512830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.512614    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-m8wgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:12.512830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.512692    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:12.676547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.676499    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:48:12.773473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.754197    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:48:12.773473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.754527    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:48:12.773473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.754743    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:48:12.850417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.849732    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" containerName="payload-pre-processing" containerID="cri-o://f9eb98bbe314c5d47e047719e7e71b8477cad8a1d6e2ed6af235ebaf5541b23b" gracePeriod=25
Nov 03 23:48:12.976299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.976251    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:12.999680    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerName="payload-processing" containerID="cri-o://db0d7b1a8e97ecd918a7d651fc1d73ed0ce1d8226501eb1c2bd5735150f2de30" gracePeriod=25
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.164875    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.143172    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.154620    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.143492    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.210797    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.210893    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.212303    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 23:48:13.231435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.212368    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 23:48:13.249248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.249184    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:13.417414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.405961    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.417414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.406776    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.497409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.493126    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.730417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.730298    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:48:13.967424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.885496    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ppkrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:13.967769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.967730    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:13.983647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:13.898141    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:48:14.059479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.057537    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.129414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.117173    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 23:48:14.190474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.190423    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:48:14.255311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.255206    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.257568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.257517    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:48:14.610979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.356583    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.611928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.459634    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.615479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.503001    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.668479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.519447    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:14.687057 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.625138    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:48:14.687656 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.687631    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:48:14.772676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.740818    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:48:14.772676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:14.741510    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.177415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.167731    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:15.265420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.262339    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:48:15.350402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.350349    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:48:15.371054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.366318    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.535506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.535379    6430 generic.go:356] "Generic (PLEG): container finished" podID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerID="db0d7b1a8e97ecd918a7d651fc1d73ed0ce1d8226501eb1c2bd5735150f2de30" exitCode=0
Nov 03 23:48:15.535814 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.535788    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" event={"ID":"2d7b4d71-a7ac-4796-8b9f-79db25db1c06","Type":"ContainerDied","Data":"db0d7b1a8e97ecd918a7d651fc1d73ed0ce1d8226501eb1c2bd5735150f2de30"}
Nov 03 23:48:15.539245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.539217    6430 generic.go:356] "Generic (PLEG): container finished" podID="771b6c87-0b69-4e64-9736-74fe3be72437" containerID="f9eb98bbe314c5d47e047719e7e71b8477cad8a1d6e2ed6af235ebaf5541b23b" exitCode=0
Nov 03 23:48:15.539678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.539653    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" event={"ID":"771b6c87-0b69-4e64-9736-74fe3be72437","Type":"ContainerDied","Data":"f9eb98bbe314c5d47e047719e7e71b8477cad8a1d6e2ed6af235ebaf5541b23b"}
Nov 03 23:48:15.591855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.591814    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.690413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.625705    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.690413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.625818    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.690413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.668488    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.738425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.692102    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:48:15.738425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.718682    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:15.738425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.720206    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:48:15.747435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.739092    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": context deadline exceeded"
Nov 03 23:48:15.764414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.747711    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:48:15.792459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.766918    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:48:15.976492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:15.976458    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:48:16.291339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.291293    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:16.318378 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.318341    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:48:16.359776 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:16.359746    6430 logging.go:55] [core] [Channel #167 SubChannel #168]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.85:50052", ServerName: "10.134.0.85:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.85:50052: connect: connection refused"
Nov 03 23:48:16.361297 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:16.361260    6430 logging.go:55] [core] [Channel #169 SubChannel #170]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.86:50052", ServerName: "10.134.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.86:50052: connect: connection refused"
Nov 03 23:48:16.403203 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.403136    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:16.693455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.693367    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:48:16.768412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.767633    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:16.791434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.778192    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:48:16.791434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.778936    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:16.791434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.780054    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:48:16.791434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.780933    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:48:16.791434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.782083    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:48:16.876178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.876131    6430 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 23:48:16.903537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.903510    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:48:16.903827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.903801    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:48:16.972851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.971172    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:48:16.972851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.972439    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:48:16.984296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.983802    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:48:16.984296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.983857    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:48:16.986788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.985875    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:48:16.989636 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:16.988706    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:48:17.073466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.073439    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:48:17.076954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.076936    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:48:17.164433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.162970    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\") pod \"771b6c87-0b69-4e64-9736-74fe3be72437\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") "
Nov 03 23:48:17.165691 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.165656    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\") pod \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") "
Nov 03 23:48:17.165840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.164771    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume" pod "771b6c87-0b69-4e64-9736-74fe3be72437" (UID: "771b6c87-0b69-4e64-9736-74fe3be72437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:48:17.165905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.165842    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\" (UniqueName: \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\") pod \"771b6c87-0b69-4e64-9736-74fe3be72437\" (UID: \"771b6c87-0b69-4e64-9736-74fe3be72437\") "
Nov 03 23:48:17.165905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.165872    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\" (UniqueName: \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\") pod \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\" (UID: \"2d7b4d71-a7ac-4796-8b9f-79db25db1c06\") "
Nov 03 23:48:17.166337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.166306    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume" pod "2d7b4d71-a7ac-4796-8b9f-79db25db1c06" (UID: "2d7b4d71-a7ac-4796-8b9f-79db25db1c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:48:17.167466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.166712    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:17.167466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.166738    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771b6c87-0b69-4e64-9736-74fe3be72437-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:17.170550 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.170525    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2" pod "771b6c87-0b69-4e64-9736-74fe3be72437" (UID: "771b6c87-0b69-4e64-9736-74fe3be72437"). InnerVolumeSpecName "kube-api-access-8l6s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:48:17.171662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.171637    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth" pod "2d7b4d71-a7ac-4796-8b9f-79db25db1c06" (UID: "2d7b4d71-a7ac-4796-8b9f-79db25db1c06"). InnerVolumeSpecName "kube-api-access-qkdth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:48:17.272211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.272084    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6s2\" (UniqueName: \"kubernetes.io/projected/771b6c87-0b69-4e64-9736-74fe3be72437-kube-api-access-8l6s2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:17.272211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.272109    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdth\" (UniqueName: \"kubernetes.io/projected/2d7b4d71-a7ac-4796-8b9f-79db25db1c06-kube-api-access-qkdth\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:48:17.369444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.367071    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.86:50052\" within 1s: context deadline exceeded"
Nov 03 23:48:17.373413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.372604    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.85:50052\" within 1s: context deadline exceeded"
Nov 03 23:48:17.373413 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:17.372811    6430 logging.go:55] [core] [Channel #169 SubChannel #170]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.86:50052", ServerName: "10.134.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.86:50052: operation was canceled"
Nov 03 23:48:17.884461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.884421    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v" event={"ID":"2d7b4d71-a7ac-4796-8b9f-79db25db1c06","Type":"ContainerDied","Data":"a1b5504cd347ca790ab67c292761509b6ac430e003b1bca1371bb52e274318d2"}
Nov 03 23:48:17.884663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.884528    6430 scope.go:122] "RemoveContainer" containerID="db0d7b1a8e97ecd918a7d651fc1d73ed0ce1d8226501eb1c2bd5735150f2de30"
Nov 03 23:48:17.884663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.884446    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"
Nov 03 23:48:17.886232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.886142    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9" event={"ID":"771b6c87-0b69-4e64-9736-74fe3be72437","Type":"ContainerDied","Data":"653dfbce6a9e655f96bda14fd51acd7c3d30c25740d95ca400848a4657970af8"}
Nov 03 23:48:17.886887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.886869    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"
Nov 03 23:48:17.896369 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.896350    6430 scope.go:122] "RemoveContainer" containerID="f9eb98bbe314c5d47e047719e7e71b8477cad8a1d6e2ed6af235ebaf5541b23b"
Nov 03 23:48:17.961410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.960843    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:48:17.968660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.968612    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9bf431-7d9f47f5c9-dh8h9"]
Nov 03 23:48:17.973538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.973507    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:48:17.979672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:17.976430    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9bf431-59c6ccc78-bpn7v"]
Nov 03 23:48:18.269873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:18.269786    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" path="/var/lib/kubelet/pods/2d7b4d71-a7ac-4796-8b9f-79db25db1c06/volumes"
Nov 03 23:48:18.270865 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:18.270745    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" path="/var/lib/kubelet/pods/771b6c87-0b69-4e64-9736-74fe3be72437/volumes"
Nov 03 23:48:20.840576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:20.840530    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:20.840976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:20.840661    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:22.295374 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.295320    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:48:22.296417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296333    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9dd01a30-7a50-4647-b306-8b014abe33e2" containerName="payload-pre-processing"
Nov 03 23:48:22.296417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296357    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd01a30-7a50-4647-b306-8b014abe33e2" containerName="payload-pre-processing"
Nov 03 23:48:22.296417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296382    6430 cpu_manager.go:414] "containerMap: removing container" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" containerName="payload-pre-processing"
Nov 03 23:48:22.296417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296408    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" containerName="payload-pre-processing"
Nov 03 23:48:22.296639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296432    6430 cpu_manager.go:414] "containerMap: removing container" podUID="41178d30-3421-4aeb-8b53-0cdd278604a0" containerName="payload-processing"
Nov 03 23:48:22.296639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296441    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41178d30-3421-4aeb-8b53-0cdd278604a0" containerName="payload-processing"
Nov 03 23:48:22.296639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296458    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerName="payload-processing"
Nov 03 23:48:22.296639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296465    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerName="payload-processing"
Nov 03 23:48:22.296821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296673    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd01a30-7a50-4647-b306-8b014abe33e2" containerName="payload-pre-processing"
Nov 03 23:48:22.296821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296687    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="771b6c87-0b69-4e64-9736-74fe3be72437" containerName="payload-pre-processing"
Nov 03 23:48:22.296821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296698    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="41178d30-3421-4aeb-8b53-0cdd278604a0" containerName="payload-processing"
Nov 03 23:48:22.296821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.296709    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7b4d71-a7ac-4796-8b9f-79db25db1c06" containerName="payload-processing"
Nov 03 23:48:22.322736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.322704    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:48:22.322912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.322839    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.336562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.333554    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-88f34dbe-serving-cert\""
Nov 03 23:48:22.464099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.464043    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.464284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.464183    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsl5q\" (UniqueName: \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.464371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.464301    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.464583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.464557    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.518625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.518580    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:48:22.541290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.541252    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:48:22.541651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.541512    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.548914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.548872    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a88e9c-openshift-default-dockercfg-24srj\""
Nov 03 23:48:22.566630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.566599    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.567356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.567328    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.567518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.567486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsl5q\" (UniqueName: \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.567597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.567555    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.568499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.568472    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.570916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.570894    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.577174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.577124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.586963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.586935    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsl5q\" (UniqueName: \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.663009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.662741    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:22.673558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.673475    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.674764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.674737    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675019 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.674999    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf52\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.675201    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.675359    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675637 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.675620    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.675800    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.675949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.675935    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.676165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.676148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.785420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.780980    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.785420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.781071    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.785420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.781107    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.785420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.781212    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.785420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.782084    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.790912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.785932    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.790912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.786034    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.790912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.786075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf52\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.790912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.786163    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.790912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.786197    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.801492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.792162    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.806674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.801956    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.817454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.807864    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.832649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.832557    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.883145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.883055    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.883900 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.883825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.907207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.907119    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:22.923611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:22.923455    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf52\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\") pod \"e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") " pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:23.270499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:23.185269    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:23.368826 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:23.368787    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:48:24.055009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:24.054969    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" event={"ID":"075ffc40-f3f9-4fe5-bd8f-e81f208a4091","Type":"ContainerStarted","Data":"fbd444cba064c5b30c953926d4d16cd4d2af0e2c54c57257041d155d11cca3ea"}
Nov 03 23:48:24.168237 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:24.168183    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f5a070_4ce8_4557_a9ad_dcee9505796e.slice/crio-b1d3e44068bb1aaa7f20c3f1cf81364a18ead4c729973b329e8ded0cfd536f22 WatchSource:0}: Error finding container b1d3e44068bb1aaa7f20c3f1cf81364a18ead4c729973b329e8ded0cfd536f22: Status 404 returned error can't find the container with id b1d3e44068bb1aaa7f20c3f1cf81364a18ead4c729973b329e8ded0cfd536f22
Nov 03 23:48:24.168473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:24.168442    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:48:24.171932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:24.171870    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:24.172032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:24.171987    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:24.172082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:24.172037    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:25.062932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.062887    6430 generic.go:356] "Generic (PLEG): container finished" podID="d32383bd-1f7a-4665-a069-732ad7b74293" containerID="29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84" exitCode=0
Nov 03 23:48:25.063511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.062959    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerDied","Data":"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"}
Nov 03 23:48:25.063625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.063605    6430 scope.go:122] "RemoveContainer" containerID="29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"
Nov 03 23:48:25.064816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.064786    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" event={"ID":"075ffc40-f3f9-4fe5-bd8f-e81f208a4091","Type":"ContainerStarted","Data":"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"}
Nov 03 23:48:25.064941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.064873    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:25.065379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.065344    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:25.065500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.065439    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:25.065500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.065487    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:25.066547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.066522    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerStarted","Data":"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"}
Nov 03 23:48:25.066658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.066567    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerStarted","Data":"b1d3e44068bb1aaa7f20c3f1cf81364a18ead4c729973b329e8ded0cfd536f22"}
Nov 03 23:48:25.269850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.186018    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:25.269850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.193871    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podStartSLOduration=3.193850708 podStartE2EDuration="3.193850708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:25.160780128 +0000 UTC m=+1121.051842065" watchObservedRunningTime="2026-11-03 23:48:25.193850708 +0000 UTC m=+1121.084912642"
Nov 03 23:48:25.269850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.240924    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:25.279528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.277538    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" podStartSLOduration=2.619077307 podStartE2EDuration="3.277519448s" totalImagesPullingTime="658.442141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:25.232766386 +0000 UTC m=+1121.123828322" watchObservedRunningTime="2026-11-03 23:48:25.277519448 +0000 UTC m=+1121.168581381"
Nov 03 23:48:25.433616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.433519    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:48:25.433933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:25.433877    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:48:26.077556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.077513    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerStarted","Data":"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"}
Nov 03 23:48:26.079269 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.079236    6430 generic.go:356] "Generic (PLEG): container finished" podID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerID="132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8" exitCode=0
Nov 03 23:48:26.079443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.079305    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerDied","Data":"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"}
Nov 03 23:48:26.079833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.079814    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:26.080542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.080102    6430 scope.go:122] "RemoveContainer" containerID="132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"
Nov 03 23:48:26.081037 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.080686    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:48:26.082077 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.082047    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:26.082157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.082131    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:26.082197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.082162    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:26.637239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.637202    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:48:26.637493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.637325    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:26.637493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:26.637372    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:27.088849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.088809    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerStarted","Data":"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"}
Nov 03 23:48:27.639430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.638738    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:27.639430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.638812    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:27.967521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.967440    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:48:27.967677 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.967535    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:27.967677 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:27.967569    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:28.628414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.627814    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:28.628414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.627900    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:28.873680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.873645    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:48:28.878247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.878218    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:28.879179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.879119    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:48:28.883484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.883459    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:48:28.885987 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.885948    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c887c7-dockercfg-8w5fb\""
Nov 03 23:48:28.886570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.886538    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c887c7\""
Nov 03 23:48:28.888570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.888554    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:48:28.888730 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.888669    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.978487    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.978725    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8nzw\" (UniqueName: \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.978775    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjl7f\" (UniqueName: \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.978840    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.981135    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:28.996414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:28.981201    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:29.141188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.138168    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8nzw\" (UniqueName: \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:29.141524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.141489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjl7f\" (UniqueName: \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:29.141741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.141725    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:29.142029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.142013    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:29.143256 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.143227    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:29.145000 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.144898    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:29.339491 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.339423    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjl7f\" (UniqueName: \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\") pod \"payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:29.360415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.352479    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8nzw\" (UniqueName: \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\") pod \"payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:29.496746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.496656    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:29.504903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.504877    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:29.741415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.730349    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:29.741415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.730429    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:29.983560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.976020    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:29.983560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:29.976081    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:30.537804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.530952    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:48:30.537804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.535635    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:48:30.635453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.633009    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:30.635453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.633067    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:30.967805 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.967721    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:30.967805 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:30.967779    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:31.086247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.086216    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:48:31.402342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.402300    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" event={"ID":"c1c751b6-d132-431c-b165-ca9b4c54cd8b","Type":"ContainerStarted","Data":"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"}
Nov 03 23:48:31.402647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.402623    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" event={"ID":"c1c751b6-d132-431c-b165-ca9b4c54cd8b","Type":"ContainerStarted","Data":"e56dada3d9f451509f35f834235ab656c67c8d66bf0089387375b00546563cee"}
Nov 03 23:48:31.403974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.403950    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:31.457255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.442331    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" podStartSLOduration=3.442308048 podStartE2EDuration="3.442308048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:31.44005162 +0000 UTC m=+1127.331113559" watchObservedRunningTime="2026-11-03 23:48:31.442308048 +0000 UTC m=+1127.333370016"
Nov 03 23:48:31.460665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.460624    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" event={"ID":"a5959b98-1daf-4ea9-bde2-3978db9c2860","Type":"ContainerStarted","Data":"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"}
Nov 03 23:48:31.465537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.465500    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" event={"ID":"a5959b98-1daf-4ea9-bde2-3978db9c2860","Type":"ContainerStarted","Data":"0dca5e0184a115f404c6d42316e5e2980f822f356ab9768f60a13d5d50717005"}
Nov 03 23:48:31.465847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.465822    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:31.496415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.491179    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" podStartSLOduration=3.491156629 podStartE2EDuration="3.491156629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:31.489815153 +0000 UTC m=+1127.380877088" watchObservedRunningTime="2026-11-03 23:48:31.491156629 +0000 UTC m=+1127.382218563"
Nov 03 23:48:31.634816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.634778    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 23:48:31.635097 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.635058    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 23:48:31.967382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.967338    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:31.967605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:31.967540    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:32.636433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.636401    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:48:32.636985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.636968    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:48:32.637612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.637589    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:48:32.743182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.743148    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:48:32.982460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.982089    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 23:48:32.982460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:32.982145    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 23:48:33.972339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:33.972306    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:48:33.972805 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:33.972669    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:48:33.973437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:33.973415    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:48:34.833238 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:48:34.833203    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:48:34.961006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:34.960974    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:48:34.964851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:34.964830    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:34.981486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:34.981462    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8cfc3c-openshift-default-dockercfg-kwhv2\""
Nov 03 23:48:34.989019 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:34.988924    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:48:35.112271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112187    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112224    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112499    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112557    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112646    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8r2p\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112716    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112815    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.112936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.112919    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215333    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215458    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215669    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215732    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215780    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r2p\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215839    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.215901    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.216698    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.217411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.216939    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.246411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.227158    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.246411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.227673    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.246411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.244967    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.260411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.253772    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.275426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.264879    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.275426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.268520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r2p\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.279426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.277725    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\") pod \"e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") " pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.293410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.290121    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:35.762618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.762594    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:48:35.763728 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:35.763699    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9042c877_e246_4e1c_90ea_81200d19e31e.slice/crio-f7b4deb73b4b34587bf8947950b3639006e35359a1244be4604e27f0539ad6d4 WatchSource:0}: Error finding container f7b4deb73b4b34587bf8947950b3639006e35359a1244be4604e27f0539ad6d4: Status 404 returned error can't find the container with id f7b4deb73b4b34587bf8947950b3639006e35359a1244be4604e27f0539ad6d4
Nov 03 23:48:35.766365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.766326    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:35.766477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.766423    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:35.766477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:35.766467    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:48:36.555461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:36.555407    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerStarted","Data":"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"}
Nov 03 23:48:36.555889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:36.555491    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerStarted","Data":"f7b4deb73b4b34587bf8947950b3639006e35359a1244be4604e27f0539ad6d4"}
Nov 03 23:48:36.590479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:36.590424    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podStartSLOduration=2.590409034 podStartE2EDuration="2.590409034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:36.58582911 +0000 UTC m=+1132.476891044" watchObservedRunningTime="2026-11-03 23:48:36.590409034 +0000 UTC m=+1132.481470968"
Nov 03 23:48:37.291979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:37.291944    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:37.297338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:37.297312    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:37.562174 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:37.562142    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:37.563139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:37.563095    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:48:38.007735 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.007645    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:48:38.011653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.011631    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.019327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.019298    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c887c7-serving-cert\""
Nov 03 23:48:38.023321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.023295    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:48:38.213457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.213410    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.213685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.213539    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.213685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.213618    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztv69\" (UniqueName: \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.214141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.213812    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.315412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.315336    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.319783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.319737    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztv69\" (UniqueName: \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.320150 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.320121    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.320378 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.320360    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.321045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.321016    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.343498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.342748    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.348855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.348827    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.351120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.350965    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztv69\" (UniqueName: \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\") pod \"maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:38.630471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:38.627585    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:39.022727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:39.022643    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:48:39.026973 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:48:39.026939    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a98d67e_ab8f_49fc_9224_d0a9a1744fdb.slice/crio-f224c55ccd59eb75ac41307114bc0f961120ed762af79aebdbae322de37584a7 WatchSource:0}: Error finding container f224c55ccd59eb75ac41307114bc0f961120ed762af79aebdbae322de37584a7: Status 404 returned error can't find the container with id f224c55ccd59eb75ac41307114bc0f961120ed762af79aebdbae322de37584a7
Nov 03 23:48:39.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:39.597670    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" event={"ID":"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb","Type":"ContainerStarted","Data":"f224c55ccd59eb75ac41307114bc0f961120ed762af79aebdbae322de37584a7"}
Nov 03 23:48:40.692414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:40.685272    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" event={"ID":"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb","Type":"ContainerStarted","Data":"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"}
Nov 03 23:48:40.692414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:40.687905    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:48:40.781337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:40.781266    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" podStartSLOduration=3.228283849 podStartE2EDuration="3.781234574s" totalImagesPullingTime="552.950725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:40.773544327 +0000 UTC m=+1136.664606275" watchObservedRunningTime="2026-11-03 23:48:40.781234574 +0000 UTC m=+1136.672296510"
Nov 03 23:48:42.485756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:42.485724    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:48:43.496852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:43.496815    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:48:47.704093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:48:47.704062    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:49:02.327830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:02.327794    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:49:02.353962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:02.353924    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:49:02.574923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:02.574886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:49:05.178447 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:05.173171    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:49:05.802570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:05.802536    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:49:07.353425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:07.343473    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" podUID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" containerName="payload-processing" containerID="cri-o://559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b" gracePeriod=25
Nov 03 23:49:07.356611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:07.356565    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" podUID="a5959b98-1daf-4ea9-bde2-3978db9c2860" containerName="payload-pre-processing" containerID="cri-o://50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143" gracePeriod=25
Nov 03 23:49:07.979331 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:07.979302    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:49:08.099685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.099661    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:49:08.177472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.172185    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\") pod \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") "
Nov 03 23:49:08.177472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.172527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\" (UniqueName: \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\") pod \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\" (UID: \"c1c751b6-d132-431c-b165-ca9b4c54cd8b\") "
Nov 03 23:49:08.184427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.184373    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume" pod "c1c751b6-d132-431c-b165-ca9b4c54cd8b" (UID: "c1c751b6-d132-431c-b165-ca9b4c54cd8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:08.189593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.184912    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw" pod "c1c751b6-d132-431c-b165-ca9b4c54cd8b" (UID: "c1c751b6-d132-431c-b165-ca9b4c54cd8b"). InnerVolumeSpecName "kube-api-access-q8nzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:08.199999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.199966    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:49:08.200654 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.200197    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" podUID="1d320f12-1741-4fe2-9064-c883ab6a12e5" containerName="maas-api" containerID="cri-o://4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6" gracePeriod=30
Nov 03 23:49:08.276428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.274163    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\" (UniqueName: \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\") pod \"a5959b98-1daf-4ea9-bde2-3978db9c2860\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") "
Nov 03 23:49:08.276428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.274217    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\") pod \"a5959b98-1daf-4ea9-bde2-3978db9c2860\" (UID: \"a5959b98-1daf-4ea9-bde2-3978db9c2860\") "
Nov 03 23:49:08.276428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.274880    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8nzw\" (UniqueName: \"kubernetes.io/projected/c1c751b6-d132-431c-b165-ca9b4c54cd8b-kube-api-access-q8nzw\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:08.276428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.274899    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c751b6-d132-431c-b165-ca9b4c54cd8b-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:08.276428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.275635    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume" pod "a5959b98-1daf-4ea9-bde2-3978db9c2860" (UID: "a5959b98-1daf-4ea9-bde2-3978db9c2860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:08.279973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.279908    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f" pod "a5959b98-1daf-4ea9-bde2-3978db9c2860" (UID: "a5959b98-1daf-4ea9-bde2-3978db9c2860"). InnerVolumeSpecName "kube-api-access-cjl7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:08.293018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.292992    6430 generic.go:356] "Generic (PLEG): container finished" podID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" containerID="559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b" exitCode=0
Nov 03 23:49:08.293260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.293054    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"
Nov 03 23:49:08.293414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.293075    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" event={"ID":"c1c751b6-d132-431c-b165-ca9b4c54cd8b","Type":"ContainerDied","Data":"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"}
Nov 03 23:49:08.293495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.293457    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb" event={"ID":"c1c751b6-d132-431c-b165-ca9b4c54cd8b","Type":"ContainerDied","Data":"e56dada3d9f451509f35f834235ab656c67c8d66bf0089387375b00546563cee"}
Nov 03 23:49:08.293554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.293492    6430 scope.go:122] "RemoveContainer" containerID="559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"
Nov 03 23:49:08.305265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.305233    6430 generic.go:356] "Generic (PLEG): container finished" podID="a5959b98-1daf-4ea9-bde2-3978db9c2860" containerID="50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143" exitCode=0
Nov 03 23:49:08.305989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.305506    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" event={"ID":"a5959b98-1daf-4ea9-bde2-3978db9c2860","Type":"ContainerDied","Data":"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"}
Nov 03 23:49:08.305989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.305584    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"
Nov 03 23:49:08.305989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.305606    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr" event={"ID":"a5959b98-1daf-4ea9-bde2-3978db9c2860","Type":"ContainerDied","Data":"0dca5e0184a115f404c6d42316e5e2980f822f356ab9768f60a13d5d50717005"}
Nov 03 23:49:08.392448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.381114    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjl7f\" (UniqueName: \"kubernetes.io/projected/a5959b98-1daf-4ea9-bde2-3978db9c2860-kube-api-access-cjl7f\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:08.392448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.381147    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959b98-1daf-4ea9-bde2-3978db9c2860-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:08.503427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.501457    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:49:08.606466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.599825    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c887c7-5c76ffbf4b-khsxb"]
Nov 03 23:49:08.627439 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.612568    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:49:08.644875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.638251    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c887c7-5c4d9df575-mhdqr"]
Nov 03 23:49:08.693589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.693564    6430 scope.go:122] "RemoveContainer" containerID="559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"
Nov 03 23:49:08.694233 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:08.694210    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b\": container with ID starting with 559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b not found: ID does not exist" containerID="559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"
Nov 03 23:49:08.705545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.705487    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b"} err="failed to get container status \"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b\": rpc error: code = NotFound desc = could not find container \"559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b\": container with ID starting with 559924f3d93aa3b68a49bba2109aadd7bbfb41f7fa5413aa5422e29f2bef608b not found: ID does not exist"
Nov 03 23:49:08.705806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.705784    6430 scope.go:122] "RemoveContainer" containerID="50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"
Nov 03 23:49:08.859599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.859572    6430 scope.go:122] "RemoveContainer" containerID="50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"
Nov 03 23:49:08.888273 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:08.888233    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143\": container with ID starting with 50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143 not found: ID does not exist" containerID="50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"
Nov 03 23:49:08.888558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:08.888523    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143"} err="failed to get container status \"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143\": rpc error: code = NotFound desc = could not find container \"50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143\": container with ID starting with 50f49d999cce11fbe4dd159d2630df22a5a8998293fe1ea9d3ab065929528143 not found: ID does not exist"
Nov 03 23:49:09.302904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.302880    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:49:09.368646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.368597    6430 generic.go:356] "Generic (PLEG): container finished" podID="1d320f12-1741-4fe2-9064-c883ab6a12e5" containerID="4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6" exitCode=0
Nov 03 23:49:09.368830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.368731    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" event={"ID":"1d320f12-1741-4fe2-9064-c883ab6a12e5","Type":"ContainerDied","Data":"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"}
Nov 03 23:49:09.368830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.368779    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6" event={"ID":"1d320f12-1741-4fe2-9064-c883ab6a12e5","Type":"ContainerDied","Data":"9af356daaf360436a5e862f13d077da20964918407bebbd00b3f21c8651c1213"}
Nov 03 23:49:09.368830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.368801    6430 scope.go:122] "RemoveContainer" containerID="4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"
Nov 03 23:49:09.368936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.368887    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"
Nov 03 23:49:09.379349 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.379329    6430 scope.go:122] "RemoveContainer" containerID="4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"
Nov 03 23:49:09.379748 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:09.379727    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6\": container with ID starting with 4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6 not found: ID does not exist" containerID="4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"
Nov 03 23:49:09.379798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.379758    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6"} err="failed to get container status \"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6\": rpc error: code = NotFound desc = could not find container \"4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6\": container with ID starting with 4855f78e5dcc21761e7e4b8dfb14551bf925d5db2912bf92ec6f0c54328647d6 not found: ID does not exist"
Nov 03 23:49:09.388377 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.388352    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\") pod \"1d320f12-1741-4fe2-9064-c883ab6a12e5\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") "
Nov 03 23:49:09.388502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.388432    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\" (UniqueName: \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\") pod \"1d320f12-1741-4fe2-9064-c883ab6a12e5\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") "
Nov 03 23:49:09.388502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.388482    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\") pod \"1d320f12-1741-4fe2-9064-c883ab6a12e5\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") "
Nov 03 23:49:09.388724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.388710    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\") pod \"1d320f12-1741-4fe2-9064-c883ab6a12e5\" (UID: \"1d320f12-1741-4fe2-9064-c883ab6a12e5\") "
Nov 03 23:49:09.389043 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.389020    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca" pod "1d320f12-1741-4fe2-9064-c883ab6a12e5" (UID: "1d320f12-1741-4fe2-9064-c883ab6a12e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:09.389336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.389317    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d320f12-1741-4fe2-9064-c883ab6a12e5-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:09.390789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.390755    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls" pod "1d320f12-1741-4fe2-9064-c883ab6a12e5" (UID: "1d320f12-1741-4fe2-9064-c883ab6a12e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:09.390990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.390971    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs" pod "1d320f12-1741-4fe2-9064-c883ab6a12e5" (UID: "1d320f12-1741-4fe2-9064-c883ab6a12e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:09.391056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.390979    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7" pod "1d320f12-1741-4fe2-9064-c883ab6a12e5" (UID: "1d320f12-1741-4fe2-9064-c883ab6a12e5"). InnerVolumeSpecName "kube-api-access-l4hv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:09.498018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.497737    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hv7\" (UniqueName: \"kubernetes.io/projected/1d320f12-1741-4fe2-9064-c883ab6a12e5-kube-api-access-l4hv7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:09.498620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.498593    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:09.498833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.498816    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d320f12-1741-4fe2-9064-c883ab6a12e5-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:09.785555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.785521    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:49:09.789732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:09.789704    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d41841-5776877d8b-qtsb6"]
Nov 03 23:49:10.266100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:10.266060    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d320f12-1741-4fe2-9064-c883ab6a12e5" path="/var/lib/kubelet/pods/1d320f12-1741-4fe2-9064-c883ab6a12e5/volumes"
Nov 03 23:49:10.266761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:10.266741    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5959b98-1daf-4ea9-bde2-3978db9c2860" path="/var/lib/kubelet/pods/a5959b98-1daf-4ea9-bde2-3978db9c2860/volumes"
Nov 03 23:49:10.267281 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:10.267263    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" path="/var/lib/kubelet/pods/c1c751b6-d132-431c-b165-ca9b4c54cd8b/volumes"
Nov 03 23:49:10.804010 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:10.803960    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" podUID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" containerName="payload-pre-processing" containerID="cri-o://2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224" gracePeriod=25
Nov 03 23:49:11.121119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.121080    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:49:11.208795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.208760    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\") pod \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") "
Nov 03 23:49:11.208961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.208877    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\" (UniqueName: \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\") pod \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\" (UID: \"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2\") "
Nov 03 23:49:11.209181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.209153    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume" pod "01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" (UID: "01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:11.209513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.209494    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:11.211367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.211337    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4" pod "01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" (UID: "01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2"). InnerVolumeSpecName "kube-api-access-2dvc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:11.311400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.311358    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvc4\" (UniqueName: \"kubernetes.io/projected/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2-kube-api-access-2dvc4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:11.485224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.485135    6430 generic.go:356] "Generic (PLEG): container finished" podID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" containerID="2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224" exitCode=0
Nov 03 23:49:11.485699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.485671    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" event={"ID":"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2","Type":"ContainerDied","Data":"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"}
Nov 03 23:49:11.485891 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.485874    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws" event={"ID":"01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2","Type":"ContainerDied","Data":"5b1195354d04d3fddd3f1b4a265f19bdf69281cc9f436d44f9ff0a6c9f109dbb"}
Nov 03 23:49:11.485999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.485988    6430 scope.go:122] "RemoveContainer" containerID="2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"
Nov 03 23:49:11.486185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.486174    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"
Nov 03 23:49:11.525279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.525256    6430 scope.go:122] "RemoveContainer" containerID="2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"
Nov 03 23:49:11.525673 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:11.525642    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224\": container with ID starting with 2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224 not found: ID does not exist" containerID="2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"
Nov 03 23:49:11.525764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.525682    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224"} err="failed to get container status \"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224\": rpc error: code = NotFound desc = could not find container \"2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224\": container with ID starting with 2deceb7de0229a9033718f497d5633b38f9754adeb0f6093825f6cfba87f9224 not found: ID does not exist"
Nov 03 23:49:11.540984 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.540954    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:49:11.544778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:11.544753    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d41841-58974f6b47-fltws"]
Nov 03 23:49:12.247229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:12.244766    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" path="/var/lib/kubelet/pods/01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2/volumes"
Nov 03 23:49:15.022597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.022557    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:49:15.023560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.023525    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" containerID="cri-o://61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26" gracePeriod=30
Nov 03 23:49:15.993414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.993223    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:49:15.997723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997686    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a5959b98-1daf-4ea9-bde2-3978db9c2860" containerName="payload-pre-processing"
Nov 03 23:49:15.997723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997719    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5959b98-1daf-4ea9-bde2-3978db9c2860" containerName="payload-pre-processing"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997783    6430 cpu_manager.go:414] "containerMap: removing container" podUID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" containerName="payload-pre-processing"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997793    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" containerName="payload-pre-processing"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997811    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" containerName="payload-processing"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997821    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" containerName="payload-processing"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997890    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1d320f12-1741-4fe2-9064-c883ab6a12e5" containerName="maas-api"
Nov 03 23:49:15.997942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.997902    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d320f12-1741-4fe2-9064-c883ab6a12e5" containerName="maas-api"
Nov 03 23:49:15.998247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.998215    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5959b98-1daf-4ea9-bde2-3978db9c2860" containerName="payload-pre-processing"
Nov 03 23:49:15.998333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.998256    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ea1ffd-9eec-47fe-ae6e-8a1ea30a92b2" containerName="payload-pre-processing"
Nov 03 23:49:15.998333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.998266    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c751b6-d132-431c-b165-ca9b4c54cd8b" containerName="payload-processing"
Nov 03 23:49:15.998333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:15.998281    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d320f12-1741-4fe2-9064-c883ab6a12e5" containerName="maas-api"
Nov 03 23:49:16.002977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.002913    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.017737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.017707    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:49:16.023288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.023259    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkt6\" (UniqueName: \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.023723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.023689    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.105455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.071849    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8cfc3c-dockercfg-bvq2d\""
Nov 03 23:49:16.105455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.072061    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8cfc3c\""
Nov 03 23:49:16.168417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.139502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkt6\" (UniqueName: \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.168417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.139847    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.168417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.140832    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.197412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.194888    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkt6\" (UniqueName: \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\") pod \"payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.323665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.323293    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:16.544452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.544421    6430 generic.go:356] "Generic (PLEG): container finished" podID="9042c877-e246-4e1c-90ea-81200d19e31e" containerID="736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a" exitCode=0
Nov 03 23:49:16.544639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.544496    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerDied","Data":"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"}
Nov 03 23:49:16.545039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.545018    6430 scope.go:122] "RemoveContainer" containerID="736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"
Nov 03 23:49:16.546176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546143    6430 generic.go:356] "Generic (PLEG): container finished" podID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerID="e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92" exitCode=0
Nov 03 23:49:16.546268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546220    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerDied","Data":"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"}
Nov 03 23:49:16.546610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546595    6430 scope.go:122] "RemoveContainer" containerID="e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"
Nov 03 23:49:16.546797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546775    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.546850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546838    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.546887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.546872    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.548009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.547981    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.548078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.548041    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.548117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.548075    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:49:16.691808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:16.691353    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:49:16.698304 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:49:16.698263    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70869874_b7ef_4a34_a97c_820f073f400d.slice/crio-ffe28b9ea752e7d8a559ee8a773ff0d470547f9537b872a0ad39f614efb62e3a WatchSource:0}: Error finding container ffe28b9ea752e7d8a559ee8a773ff0d470547f9537b872a0ad39f614efb62e3a: Status 404 returned error can't find the container with id ffe28b9ea752e7d8a559ee8a773ff0d470547f9537b872a0ad39f614efb62e3a
Nov 03 23:49:17.585959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:17.585917    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerStarted","Data":"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"}
Nov 03 23:49:17.594582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:17.593812    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" event={"ID":"70869874-b7ef-4a34-a97c-820f073f400d","Type":"ContainerStarted","Data":"ffe28b9ea752e7d8a559ee8a773ff0d470547f9537b872a0ad39f614efb62e3a"}
Nov 03 23:49:17.597615 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:17.597580    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerStarted","Data":"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"}
Nov 03 23:49:17.673419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:17.666685    6430 patch_prober.go:29] interesting pod/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:49:17.673419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:17.666783    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:49:18.186352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.185908    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:18.186352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.186054    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:18.186352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.186108    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:18.309183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.309143    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:18.309495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.309464    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:18.311287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:18.311255    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:49:19.186951 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:19.186912    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:19.187342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:19.186978    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:19.291798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:19.291761    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:19.292001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:19.291831    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:20.186707 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.186666    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:20.186874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.186735    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:20.292282 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.292248    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:20.292821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.292786    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:20.494258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.493974    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:49:20.585914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.585876    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586010    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586069    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586100    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586139    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586184    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586237    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586313    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.586517 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.586505    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\") pod \"d32383bd-1f7a-4665-a069-732ad7b74293\" (UID: \"d32383bd-1f7a-4665-a069-732ad7b74293\") "
Nov 03 23:49:20.587628 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.587357    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:20.587782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.587720    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:20.587782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.587758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:20.588162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.588015    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:20.588162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.588137    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:20.588923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.588899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:49:20.592996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.592571    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:20.592996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.592959    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "kube-api-access-xlhj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:20.597021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.596823    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token" pod "d32383bd-1f7a-4665-a069-732ad7b74293" (UID: "d32383bd-1f7a-4665-a069-732ad7b74293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.698979    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlhj5\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-kube-api-access-xlhj5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699021    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32383bd-1f7a-4665-a069-732ad7b74293-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699037    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699051    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32383bd-1f7a-4665-a069-732ad7b74293-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699065    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32383bd-1f7a-4665-a069-732ad7b74293-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699078    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699091    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699104    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.738907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.699118    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32383bd-1f7a-4665-a069-732ad7b74293-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:20.828428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.827008    6430 generic.go:356] "Generic (PLEG): container finished" podID="d32383bd-1f7a-4665-a069-732ad7b74293" containerID="61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26" exitCode=0
Nov 03 23:49:20.828428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.827217    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerDied","Data":"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"}
Nov 03 23:49:20.828428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.827292    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft" event={"ID":"d32383bd-1f7a-4665-a069-732ad7b74293","Type":"ContainerDied","Data":"60ef6fedd3198ecc542fc51946a6a075c2fbb22039e3167c258ec1ecd3b5f8c5"}
Nov 03 23:49:20.828428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.827345    6430 scope.go:122] "RemoveContainer" containerID="61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"
Nov 03 23:49:20.828428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.827737    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"
Nov 03 23:49:20.921574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.920566    6430 scope.go:122] "RemoveContainer" containerID="29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"
Nov 03 23:49:20.977770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.977629    6430 scope.go:122] "RemoveContainer" containerID="61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"
Nov 03 23:49:20.978534 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:20.978341    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26\": container with ID starting with 61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26 not found: ID does not exist" containerID="61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"
Nov 03 23:49:20.978534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.978398    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26"} err="failed to get container status \"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26\": rpc error: code = NotFound desc = could not find container \"61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26\": container with ID starting with 61014539c9f31f5404fdc32dc8b7f328b367cb28cb7490538dd1e2d2b7906d26 not found: ID does not exist"
Nov 03 23:49:20.978534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.978423    6430 scope.go:122] "RemoveContainer" containerID="29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"
Nov 03 23:49:20.979229 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:20.979201    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84\": container with ID starting with 29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84 not found: ID does not exist" containerID="29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"
Nov 03 23:49:20.979611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.979567    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:49:20.979611 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.979591    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84"} err="failed to get container status \"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84\": rpc error: code = NotFound desc = could not find container \"29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84\": container with ID starting with 29b32bf1f10d449116a855f53804ac1e5294641c2472f551d35b75efe45bbe84 not found: ID does not exist"
Nov 03 23:49:20.987804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:20.987658    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d41841-openshift-default-d6d57f569-xlmft"]
Nov 03 23:49:21.186356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:21.186245    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:21.186356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:21.186304    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:21.291675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:21.291563    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:21.291675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:21.291626    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:22.187884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.186374    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:22.187884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.186444    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:22.294633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.294158    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:22.294633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.294215    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:22.298981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.298950    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" path="/var/lib/kubelet/pods/d32383bd-1f7a-4665-a069-732ad7b74293/volumes"
Nov 03 23:49:22.568510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.568471    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:49:22.882753 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.882630    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" event={"ID":"70869874-b7ef-4a34-a97c-820f073f400d","Type":"ContainerStarted","Data":"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"}
Nov 03 23:49:22.884587 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.884563    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:22.934006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:22.929015    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" podStartSLOduration=2.462526651 podStartE2EDuration="7.92899109s" totalImagesPullingTime="5.466464439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:22.913350599 +0000 UTC m=+1178.804412534" watchObservedRunningTime="2026-11-03 23:49:22.92899109 +0000 UTC m=+1178.820053025"
Nov 03 23:49:23.189527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:23.188941    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 23:49:23.189527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:23.189006    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 23:49:23.330055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:23.330000    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 23:49:23.330353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:23.330322    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 23:49:24.219242 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.219209    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:24.219799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.219777    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:24.221108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.221082    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:24.296861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.296833    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:49:24.298035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.298014    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:49:24.574601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:24.574564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:49:25.433537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.433497    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:49:25.434561 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434524    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.434740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434715    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.434740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434743    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.434903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434751    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.434979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434964    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.435035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.434982    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32383bd-1f7a-4665-a069-732ad7b74293" containerName="istio-proxy"
Nov 03 23:49:25.439104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.439084    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.447180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.447151    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e4892d\"/\"default-dockercfg-h86kn\""
Nov 03 23:49:25.449227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.449203    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:49:25.449965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.449940    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e4892d\"/\"openshift-service-ca.crt\""
Nov 03 23:49:25.449965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.449957    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e4892d\"/\"kube-root-ca.crt\""
Nov 03 23:49:25.499693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.499658    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e4892d\"/\"ipp-route-e4892d-kserve-self-signed-certs\""
Nov 03 23:49:25.508700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.508667    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.508864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.508741    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.508864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.508843    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.508985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.508881    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.508985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.508900    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.509092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.509051    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqwl\" (UniqueName: \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610048    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610092    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610173    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqwl\" (UniqueName: \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610304    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610364    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610415    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610525    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610700    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.610784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.610766    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.612323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.612300    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.612658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.612641    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.624366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.624314    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqwl\" (UniqueName: \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\") pod \"ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") " pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:25.764509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:25.761677    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:26.183201 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:49:26.183167    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e82018a_1491_47aa_b218_68a35f1de35c.slice/crio-dd9b086526f037731d9085c360b6e181b50660adfbec37b37a8ac4ba652161e3 WatchSource:0}: Error finding container dd9b086526f037731d9085c360b6e181b50660adfbec37b37a8ac4ba652161e3: Status 404 returned error can't find the container with id dd9b086526f037731d9085c360b6e181b50660adfbec37b37a8ac4ba652161e3
Nov 03 23:49:26.183336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:26.183298    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:49:26.915382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:26.915339    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" event={"ID":"8e82018a-1491-47aa-b218-68a35f1de35c","Type":"ContainerStarted","Data":"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"}
Nov 03 23:49:26.915884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:26.915446    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" event={"ID":"8e82018a-1491-47aa-b218-68a35f1de35c","Type":"ContainerStarted","Data":"dd9b086526f037731d9085c360b6e181b50660adfbec37b37a8ac4ba652161e3"}
Nov 03 23:49:26.942310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:26.942255    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" podStartSLOduration=1.942242453 podStartE2EDuration="1.942242453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:26.936779149 +0000 UTC m=+1182.827841084" watchObservedRunningTime="2026-11-03 23:49:26.942242453 +0000 UTC m=+1182.833304387"
Nov 03 23:49:27.769415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:27.769232    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:34.831991 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:34.831956    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:49:34.899884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:34.899850    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:49:35.765795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:35.765760    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:35.778430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:35.778402    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:36.071269 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:36.071242    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:49:39.215905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:39.215869    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:49:39.216573 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:39.216130    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" containerID="cri-o://36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9" gracePeriod=30
Nov 03 23:49:39.223613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:39.223580    6430 patch_prober.go:29] interesting pod/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:49:39.223744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:39.223638    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:49:44.220799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.220764    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:49:44.221250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.220963    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" podUID="02a11c94-9578-4e43-b02a-7feee8281935" containerName="maas-api" containerID="cri-o://ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc" gracePeriod=30
Nov 03 23:49:44.919516 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.919478    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:49:44.922045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.922021    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:44.991567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.991727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.991727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991702    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.991856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991758    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.991856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991790    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\") pod \"02a11c94-9578-4e43-b02a-7feee8281935\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") "
Nov 03 23:49:44.991967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.991967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991955    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\") pod \"02a11c94-9578-4e43-b02a-7feee8281935\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") "
Nov 03 23:49:44.992069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991959    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:44.992069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.991980    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:44.992069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992014    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.992069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992064    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.992275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992097    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:44.992275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992157    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\" (UniqueName: \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\") pod \"02a11c94-9578-4e43-b02a-7feee8281935\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") "
Nov 03 23:49:44.992275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992250    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\") pod \"02a11c94-9578-4e43-b02a-7feee8281935\" (UID: \"02a11c94-9578-4e43-b02a-7feee8281935\") "
Nov 03 23:49:44.992467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992306    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.992467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.992461    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\") pod \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\" (UID: \"f4f5a070-4ce8-4557-a9ad-dcee9505796e\") "
Nov 03 23:49:44.993072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993043    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca" pod "02a11c94-9578-4e43-b02a-7feee8281935" (UID: "02a11c94-9578-4e43-b02a-7feee8281935"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:44.993233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993208    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:44.993297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993240    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a11c94-9578-4e43-b02a-7feee8281935-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:44.993297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993257    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:44.993297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993270    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:44.993476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993327    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:44.993987 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.993960    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:44.996163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.995373    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "kube-api-access-txf52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:44.996163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.995482    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs" pod "02a11c94-9578-4e43-b02a-7feee8281935" (UID: "02a11c94-9578-4e43-b02a-7feee8281935"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:44.996708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.996687    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:44.996802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.996773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:49:44.996995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.996962    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw" pod "02a11c94-9578-4e43-b02a-7feee8281935" (UID: "02a11c94-9578-4e43-b02a-7feee8281935"). InnerVolumeSpecName "kube-api-access-sggsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:44.997120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.997029    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy" pod "f4f5a070-4ce8-4557-a9ad-dcee9505796e" (UID: "f4f5a070-4ce8-4557-a9ad-dcee9505796e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:44.998275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:44.998248    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls" pod "02a11c94-9578-4e43-b02a-7feee8281935" (UID: "02a11c94-9578-4e43-b02a-7feee8281935"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:45.094442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094403    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094433    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txf52\" (UniqueName: \"kubernetes.io/projected/f4f5a070-4ce8-4557-a9ad-dcee9505796e-kube-api-access-txf52\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094445    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggsw\" (UniqueName: \"kubernetes.io/projected/02a11c94-9578-4e43-b02a-7feee8281935-kube-api-access-sggsw\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094456    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094467    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094481    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094491    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094501    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f5a070-4ce8-4557-a9ad-dcee9505796e-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.094653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.094509    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a11c94-9578-4e43-b02a-7feee8281935-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:45.153856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.153759    6430 generic.go:356] "Generic (PLEG): container finished" podID="02a11c94-9578-4e43-b02a-7feee8281935" containerID="ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc" exitCode=0
Nov 03 23:49:45.153856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.153814    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"
Nov 03 23:49:45.153856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.153838    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" event={"ID":"02a11c94-9578-4e43-b02a-7feee8281935","Type":"ContainerDied","Data":"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"}
Nov 03 23:49:45.154091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.153911    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2" event={"ID":"02a11c94-9578-4e43-b02a-7feee8281935","Type":"ContainerDied","Data":"26ab7e8d34f8fc54780347a897b64a4eace9eed4c733c9462b4605970c3248bd"}
Nov 03 23:49:45.154091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.153934    6430 scope.go:122] "RemoveContainer" containerID="ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"
Nov 03 23:49:45.155953 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.155924    6430 generic.go:356] "Generic (PLEG): container finished" podID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerID="36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9" exitCode=0
Nov 03 23:49:45.156070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.155964    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"
Nov 03 23:49:45.156070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.155977    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerDied","Data":"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"}
Nov 03 23:49:45.156070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.156020    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv" event={"ID":"f4f5a070-4ce8-4557-a9ad-dcee9505796e","Type":"ContainerDied","Data":"b1d3e44068bb1aaa7f20c3f1cf81364a18ead4c729973b329e8ded0cfd536f22"}
Nov 03 23:49:45.166351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.166331    6430 scope.go:122] "RemoveContainer" containerID="ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"
Nov 03 23:49:45.166726 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:45.166702    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc\": container with ID starting with ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc not found: ID does not exist" containerID="ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"
Nov 03 23:49:45.166788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.166737    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc"} err="failed to get container status \"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc\": rpc error: code = NotFound desc = could not find container \"ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc\": container with ID starting with ae3e5443ad04a9be0be0eed307f0e96c1d9bb66e17795f89edcc2660a9ff9bdc not found: ID does not exist"
Nov 03 23:49:45.166788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.166761    6430 scope.go:122] "RemoveContainer" containerID="36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"
Nov 03 23:49:45.177441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.177424    6430 scope.go:122] "RemoveContainer" containerID="e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"
Nov 03 23:49:45.195535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.195506    6430 scope.go:122] "RemoveContainer" containerID="36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"
Nov 03 23:49:45.198024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.197997    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:49:45.202294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.202267    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bf431-6bf756596d-4ctx2"]
Nov 03 23:49:45.235529 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:45.223085    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9\": container with ID starting with 36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9 not found: ID does not exist" containerID="36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"
Nov 03 23:49:45.235529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.223134    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9"} err="failed to get container status \"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9\": rpc error: code = NotFound desc = could not find container \"36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9\": container with ID starting with 36cd754f2692b9de07c352ea879d5bd81105e2ba7e4c6a7940208cd4ddcdf5b9 not found: ID does not exist"
Nov 03 23:49:45.235529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.223162    6430 scope.go:122] "RemoveContainer" containerID="e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"
Nov 03 23:49:45.238604 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:45.238562    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92\": container with ID starting with e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92 not found: ID does not exist" containerID="e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"
Nov 03 23:49:45.238888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.238861    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92"} err="failed to get container status \"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92\": rpc error: code = NotFound desc = could not find container \"e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92\": container with ID starting with e794e025850839643cbf095776d3141a429b16d41902602e7c90d37970792d92 not found: ID does not exist"
Nov 03 23:49:45.328180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.327463    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:49:45.357298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:45.357073    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a88e9c-openshift-default-7445767fdc-ffmcv"]
Nov 03 23:49:46.233198 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.233157    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a11c94-9578-4e43-b02a-7feee8281935" path="/var/lib/kubelet/pods/02a11c94-9578-4e43-b02a-7feee8281935/volumes"
Nov 03 23:49:46.233780 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.233759    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" path="/var/lib/kubelet/pods/f4f5a070-4ce8-4557-a9ad-dcee9505796e/volumes"
Nov 03 23:49:46.413034 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.412990    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:49:46.413821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413798    6430 cpu_manager.go:414] "containerMap: removing container" podUID="02a11c94-9578-4e43-b02a-7feee8281935" containerName="maas-api"
Nov 03 23:49:46.413921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413822    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a11c94-9578-4e43-b02a-7feee8281935" containerName="maas-api"
Nov 03 23:49:46.413921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413849    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.413921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413856    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.413921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413888    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.413921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.413895    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.414163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.414083    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a11c94-9578-4e43-b02a-7feee8281935" containerName="maas-api"
Nov 03 23:49:46.414163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.414099    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.414163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.414111    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f5a070-4ce8-4557-a9ad-dcee9505796e" containerName="istio-proxy"
Nov 03 23:49:46.419554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.419529    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.425761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.425737    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c7e425\""
Nov 03 23:49:46.426619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.426592    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c7e425-dockercfg-2hwl6\""
Nov 03 23:49:46.427293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.427273    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:49:46.436008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.435976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjg87\" (UniqueName: \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.436167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.436140    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.474793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.474759    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:49:46.480907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.480301    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.486593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.486531    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:49:46.582412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.571671    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjg87\" (UniqueName: \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.582412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.571833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.616589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.615479    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:46.702409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.700264    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zvp\" (UniqueName: \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.702409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.700374    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.809412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.803640    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zvp\" (UniqueName: \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.809412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.803773    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.809412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.805162    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.850559 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.831173    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zvp\" (UniqueName: \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\") pod \"payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:46.866794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:46.859908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjg87\" (UniqueName: \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\") pod \"payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:47.035947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:47.035904    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:47.096268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:47.096237    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:47.340351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:47.340196    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:49:47.402851 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:47.402826    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:49:47.408777 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:49:47.408750    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee8f77_08fc_4fb3_a3d3_2adc4cc86ec8.slice/crio-fb5a2ec27a4fe2cbd79e745be9436bc99eb0508f748ac231ad0c87c07b248afb WatchSource:0}: Error finding container fb5a2ec27a4fe2cbd79e745be9436bc99eb0508f748ac231ad0c87c07b248afb: Status 404 returned error can't find the container with id fb5a2ec27a4fe2cbd79e745be9436bc99eb0508f748ac231ad0c87c07b248afb
Nov 03 23:49:48.204875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.204834    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" event={"ID":"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8","Type":"ContainerStarted","Data":"bcd710681759de6f5c21c0f6e162321f9c6a499bde5d0b1dbff86764b4008357"}
Nov 03 23:49:48.205355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.204932    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:48.205355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.204956    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" event={"ID":"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8","Type":"ContainerStarted","Data":"fb5a2ec27a4fe2cbd79e745be9436bc99eb0508f748ac231ad0c87c07b248afb"}
Nov 03 23:49:48.206931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.206904    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" event={"ID":"2b39a41c-dde8-412e-947e-ad2ec0a37477","Type":"ContainerStarted","Data":"a64f5d5f5283b7e711cfa12cfd701feaf19738baf779c56c6f7e1a2f3ad9b36e"}
Nov 03 23:49:48.207055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.206946    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" event={"ID":"2b39a41c-dde8-412e-947e-ad2ec0a37477","Type":"ContainerStarted","Data":"8688d48b649fba34aa1543f9065d4e6803863137e8dff9c54e19e809d82fafe5"}
Nov 03 23:49:48.207055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.206996    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:48.230498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.230448    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" podStartSLOduration=2.230434453 podStartE2EDuration="2.230434453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:48.227058617 +0000 UTC m=+1204.118120552" watchObservedRunningTime="2026-11-03 23:49:48.230434453 +0000 UTC m=+1204.121496387"
Nov 03 23:49:48.258001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:48.257939    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" podStartSLOduration=2.257920976 podStartE2EDuration="2.257920976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:48.252807713 +0000 UTC m=+1204.143869649" watchObservedRunningTime="2026-11-03 23:49:48.257920976 +0000 UTC m=+1204.148982912"
Nov 03 23:49:53.185784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:53.185748    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:49:53.186290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:53.185999    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy" containerID="cri-o://0f01b975580cfd4e20699fb9828f67302343ce6855dcf33eb96304d9922d0dad" gracePeriod=30
Nov 03 23:49:55.606479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.606443    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:49:55.617526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.617498    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.621079 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.621025    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:49:55.626569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.626546    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c7e425-serving-cert\""
Nov 03 23:49:55.772522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.772467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxg4\" (UniqueName: \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.772897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.772867    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.773368 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.773345    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.773634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.773616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.876464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.875863    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.876464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.876234    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.876464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.876324    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.876464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.876436    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxg4\" (UniqueName: \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.878305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.878278    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.882628 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.882602    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.882801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.882635    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.894221 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.894187    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxg4\" (UniqueName: \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\") pod \"maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:55.942691 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:55.942637    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:56.372409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:56.364768    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:49:56.389813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:56.384815    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:49:56.389813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:56.385059    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" podUID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" containerName="maas-api" containerID="cri-o://d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2" gracePeriod=30
Nov 03 23:49:57.113698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.113669    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:49:57.259420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.258984    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\") pod \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") "
Nov 03 23:49:57.259420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.259125    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\" (UniqueName: \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\") pod \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") "
Nov 03 23:49:57.259420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.259406    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\") pod \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") "
Nov 03 23:49:57.259854 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.259471    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\") pod \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\" (UID: \"075ffc40-f3f9-4fe5-bd8f-e81f208a4091\") "
Nov 03 23:49:57.263487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.263067    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca" pod "075ffc40-f3f9-4fe5-bd8f-e81f208a4091" (UID: "075ffc40-f3f9-4fe5-bd8f-e81f208a4091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:57.266013 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.265983    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs" pod "075ffc40-f3f9-4fe5-bd8f-e81f208a4091" (UID: "075ffc40-f3f9-4fe5-bd8f-e81f208a4091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:57.266181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.266147    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q" pod "075ffc40-f3f9-4fe5-bd8f-e81f208a4091" (UID: "075ffc40-f3f9-4fe5-bd8f-e81f208a4091"). InnerVolumeSpecName "kube-api-access-hsl5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:57.266490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.266465    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls" pod "075ffc40-f3f9-4fe5-bd8f-e81f208a4091" (UID: "075ffc40-f3f9-4fe5-bd8f-e81f208a4091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:49:57.363239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.363157    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:57.363239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.363188    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:57.363239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.363197    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:57.363239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.363206    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsl5q\" (UniqueName: \"kubernetes.io/projected/075ffc40-f3f9-4fe5-bd8f-e81f208a4091-kube-api-access-hsl5q\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:57.399034 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.398998    6430 generic.go:356] "Generic (PLEG): container finished" podID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" containerID="d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2" exitCode=0
Nov 03 23:49:57.399193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.399048    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"
Nov 03 23:49:57.399193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.399090    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" event={"ID":"075ffc40-f3f9-4fe5-bd8f-e81f208a4091","Type":"ContainerDied","Data":"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"}
Nov 03 23:49:57.399193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.399143    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l" event={"ID":"075ffc40-f3f9-4fe5-bd8f-e81f208a4091","Type":"ContainerDied","Data":"fbd444cba064c5b30c953926d4d16cd4d2af0e2c54c57257041d155d11cca3ea"}
Nov 03 23:49:57.399193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.399165    6430 scope.go:122] "RemoveContainer" containerID="d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"
Nov 03 23:49:57.401369 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.401341    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" event={"ID":"dae26483-6b7a-4e00-9cae-7a8d25a250c7","Type":"ContainerStarted","Data":"5f8da1a02ea59bf6aecc192d6fe6b720dd7ce2e989a287218e3e70badcc239e8"}
Nov 03 23:49:57.401535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.401412    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" event={"ID":"dae26483-6b7a-4e00-9cae-7a8d25a250c7","Type":"ContainerStarted","Data":"21b1b1913762d3a8eabb0f414f23ef64b171e373b0ead262916d8bb3312d9795"}
Nov 03 23:49:57.401535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.401448    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:49:57.410552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.410532    6430 scope.go:122] "RemoveContainer" containerID="d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"
Nov 03 23:49:57.410846 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:49:57.410829    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2\": container with ID starting with d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2 not found: ID does not exist" containerID="d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"
Nov 03 23:49:57.410941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.410856    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2"} err="failed to get container status \"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2\": rpc error: code = NotFound desc = could not find container \"d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2\": container with ID starting with d052c574c896e8b33b1fea906edce43a8e631b6556f0b250cb61eef2201be2f2 not found: ID does not exist"
Nov 03 23:49:57.426133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.426092    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" podStartSLOduration=1.7456463009999998 podStartE2EDuration="2.426079425s" totalImagesPullingTime="680.433124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:57.42544399 +0000 UTC m=+1213.316505934" watchObservedRunningTime="2026-11-03 23:49:57.426079425 +0000 UTC m=+1213.317141359"
Nov 03 23:49:57.445471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.445402    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:49:57.449094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:57.449069    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-7kw8l"]
Nov 03 23:49:58.235283 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.235250    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" path="/var/lib/kubelet/pods/075ffc40-f3f9-4fe5-bd8f-e81f208a4091/volumes"
Nov 03 23:49:58.425712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.425674    6430 generic.go:356] "Generic (PLEG): container finished" podID="960332d4-0549-4a97-9573-c8204aec94ab" containerID="0f01b975580cfd4e20699fb9828f67302343ce6855dcf33eb96304d9922d0dad" exitCode=0
Nov 03 23:49:58.425898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.425756    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerDied","Data":"0f01b975580cfd4e20699fb9828f67302343ce6855dcf33eb96304d9922d0dad"}
Nov 03 23:49:58.425898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.425852    6430 scope.go:122] "RemoveContainer" containerID="672c78192b631c0f953db5db44b29292dfd623135a898bf952803cba000cb13a"
Nov 03 23:49:58.481078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.481013    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:49:58.653041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653011    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653104    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653222 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653203    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653269    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653427    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653719    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.653797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\") pod \"960332d4-0549-4a97-9573-c8204aec94ab\" (UID: \"960332d4-0549-4a97-9573-c8204aec94ab\") "
Nov 03 23:49:58.654014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653383    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:58.654094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653607    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:58.654094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653777    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:58.654094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.653994    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:49:58.654588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.654562    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:58.654588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.654575    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960332d4-0549-4a97-9573-c8204aec94ab-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.654719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.654610    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.654719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.654625    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.654719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.654639    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.655958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.655930    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "kube-api-access-2vf6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:58.656139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.656118    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:49:58.656445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.656372    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:49:58.656552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.656534    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy" pod "960332d4-0549-4a97-9573-c8204aec94ab" (UID: "960332d4-0549-4a97-9573-c8204aec94ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:49:58.756068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.755978    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.756068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.756008    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.756068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.756022    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960332d4-0549-4a97-9573-c8204aec94ab-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.756068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.756036    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vf6s\" (UniqueName: \"kubernetes.io/projected/960332d4-0549-4a97-9573-c8204aec94ab-kube-api-access-2vf6s\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:58.756068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:58.756050    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960332d4-0549-4a97-9573-c8204aec94ab-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:49:59.249733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.249699    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:49:59.250249 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.249905    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:49:59.434768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.434736    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"
Nov 03 23:49:59.434918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.434771    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n" event={"ID":"960332d4-0549-4a97-9573-c8204aec94ab","Type":"ContainerDied","Data":"fdfe48857d5f2bc7fa5c4eed2c333e749cf0c8be7fd56c44277a0280be47b7dc"}
Nov 03 23:49:59.434918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.434835    6430 scope.go:122] "RemoveContainer" containerID="0f01b975580cfd4e20699fb9828f67302343ce6855dcf33eb96304d9922d0dad"
Nov 03 23:49:59.471654 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.471619    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:49:59.482840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:49:59.482807    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9bf431-openshift-default-758b89f977-28v2n"]
Nov 03 23:50:00.091990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:00.089806    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:50:00.091990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:00.090024    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy" containerID="cri-o://57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2" gracePeriod=30
Nov 03 23:50:00.318413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:00.318041    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960332d4-0549-4a97-9573-c8204aec94ab" path="/var/lib/kubelet/pods/960332d4-0549-4a97-9573-c8204aec94ab/volumes"
Nov 03 23:50:01.377785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.377750    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:50:01.378546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378527    6430 cpu_manager.go:414] "containerMap: removing container" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:01.378546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378546    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:01.378653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378586    6430 cpu_manager.go:414] "containerMap: removing container" podUID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" containerName="maas-api"
Nov 03 23:50:01.378653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378591    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" containerName="maas-api"
Nov 03 23:50:01.378723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378675    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="075ffc40-f3f9-4fe5-bd8f-e81f208a4091" containerName="maas-api"
Nov 03 23:50:01.378723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378691    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:01.378723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.378701    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:01.382362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.382320    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.398925 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.398427    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-532e80-openshift-default-dockercfg-2mqht\""
Nov 03 23:50:01.402375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.402346    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.485909    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.486514    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.486591    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zq6\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.486673    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.486716    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.486749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.486739    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.495234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.488487    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.495234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.488627    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.495234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.488782    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.596683    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.596815    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.596938    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.596997    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597030    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597075    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zq6\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597148    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597182    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.597705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597204    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.598303 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.597756    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.598303 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.598193    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.614417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.599231    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.614417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.613404    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.618411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.617692    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.685968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.685891    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.686858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.686828    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.718409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.689942    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:01.719314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:01.719280    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zq6\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\") pod \"e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") " pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:02.001622 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.001540    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:02.329652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.329621    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:50:02.329764 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:02.329629    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23b7462_87c4_403d_baa4_7f6bc47ac435.slice/crio-cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb WatchSource:0}: Error finding container cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb: Status 404 returned error can't find the container with id cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb
Nov 03 23:50:02.332299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.332266    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:02.332404 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.332327    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:02.332404 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.332355    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:02.475820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.475775    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerStarted","Data":"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"}
Nov 03 23:50:02.475820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.475828    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerStarted","Data":"cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb"}
Nov 03 23:50:02.503978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:02.503913    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podStartSLOduration=1.503896271 podStartE2EDuration="1.503896271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:02.500340261 +0000 UTC m=+1218.391402233" watchObservedRunningTime="2026-11-03 23:50:02.503896271 +0000 UTC m=+1218.394958209"
Nov 03 23:50:03.001869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.001833    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:03.007875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.007846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:03.433208 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.433177    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:50:03.483749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.483718    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:03.484864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.484841    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:03.626884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:03.626846    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:50:04.831595 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:04.831555    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:50:05.372597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.372530    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.483699    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.483762    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.483842    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.483917    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.484072    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.484131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.484180    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.484222    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.484289    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\") pod \"814cf5ad-780e-49e2-96b8-453bf82698be\" (UID: \"814cf5ad-780e-49e2-96b8-453bf82698be\") "
Nov 03 23:50:05.509700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.491083    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.510967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.510567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.510967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.510754    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.510967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.510817    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.518197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.513345    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:05.518197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.517532    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "kube-api-access-4rff2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:05.522059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.521980    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:05.523001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.522966    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:05.534004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.533954    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo" pod "814cf5ad-780e-49e2-96b8-453bf82698be" (UID: "814cf5ad-780e-49e2-96b8-453bf82698be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:05.559463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.557907    6430 generic.go:356] "Generic (PLEG): container finished" podID="814cf5ad-780e-49e2-96b8-453bf82698be" containerID="57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2" exitCode=0
Nov 03 23:50:05.559463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.559194    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"
Nov 03 23:50:05.559673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.559480    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" event={"ID":"814cf5ad-780e-49e2-96b8-453bf82698be","Type":"ContainerDied","Data":"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"}
Nov 03 23:50:05.559673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.559560    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77" event={"ID":"814cf5ad-780e-49e2-96b8-453bf82698be","Type":"ContainerDied","Data":"85668b75bfd3c837ecedba2c4f8043bd8cc092fb31d07031f3bafc9e5925a40d"}
Nov 03 23:50:05.559673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.559592    6430 scope.go:122] "RemoveContainer" containerID="57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"
Nov 03 23:50:05.576716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.576685    6430 scope.go:122] "RemoveContainer" containerID="57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"
Nov 03 23:50:05.577995 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:05.577964    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2\": container with ID starting with 57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2 not found: ID does not exist" containerID="57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"
Nov 03 23:50:05.578096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.578006    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2"} err="failed to get container status \"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2\": rpc error: code = NotFound desc = could not find container \"57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2\": container with ID starting with 57e5e815f4565211f10292a68956a8c11ad8964262e6ae787142d51b3f8497d2 not found: ID does not exist"
Nov 03 23:50:05.587960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.587934    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588111    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588195    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588282 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588272    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588346    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/814cf5ad-780e-49e2-96b8-453bf82698be-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588440    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588524    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/814cf5ad-780e-49e2-96b8-453bf82698be-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588600    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/814cf5ad-780e-49e2-96b8-453bf82698be-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.588695 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.588685    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rff2\" (UniqueName: \"kubernetes.io/projected/814cf5ad-780e-49e2-96b8-453bf82698be-kube-api-access-4rff2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:05.610714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.610686    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:50:05.615030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:05.615004    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-85w77"]
Nov 03 23:50:06.211637 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.211606    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:50:06.212292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.212273    6430 cpu_manager.go:414] "containerMap: removing container" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:06.212292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.212290    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960332d4-0549-4a97-9573-c8204aec94ab" containerName="istio-proxy"
Nov 03 23:50:06.212525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.212338    6430 cpu_manager.go:414] "containerMap: removing container" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy"
Nov 03 23:50:06.212525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.212345    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy"
Nov 03 23:50:06.212625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.212531    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" containerName="istio-proxy"
Nov 03 23:50:06.218114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.218090    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:50:06.218271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.218225    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.226929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.223227    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.226929 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.224097    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:50:06.233353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.231805    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-532e80\""
Nov 03 23:50:06.235232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.235198    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-532e80-dockercfg-6kqcj\""
Nov 03 23:50:06.241972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.241479    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814cf5ad-780e-49e2-96b8-453bf82698be" path="/var/lib/kubelet/pods/814cf5ad-780e-49e2-96b8-453bf82698be/volumes"
Nov 03 23:50:06.241972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.241894    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:50:06.353411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.348679    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.353411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.348744    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftgx\" (UniqueName: \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.356486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.353820    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.392610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.387681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pfp\" (UniqueName: \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.499485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.493943    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.499485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.494003    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pfp\" (UniqueName: \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.499485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.494081    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.499485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.494113    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftgx\" (UniqueName: \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.499485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.495424    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.512446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.496373    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.524456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.524418    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pfp\" (UniqueName: \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\") pod \"payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:06.538363 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.538330    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftgx\" (UniqueName: \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\") pod \"payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.548834 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.548795    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:06.558318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:06.558291    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:07.020429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.018437    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:50:07.084471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.074310    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:50:07.586139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.586100    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" event={"ID":"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e","Type":"ContainerStarted","Data":"7777702e086fe9f2dbdd8694b798ade95ba36933dc139e5bda9297e9929f3b09"}
Nov 03 23:50:07.586624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.586163    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" event={"ID":"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e","Type":"ContainerStarted","Data":"b3c60e2e1f4a837f9e7bf56f6362d5b67abf870169ef675842ea7bdd8548c146"}
Nov 03 23:50:07.586624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.586193    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:07.587965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.587932    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" event={"ID":"211ddd11-9302-4c85-9330-0269815f2af4","Type":"ContainerStarted","Data":"9de4ecd8c05f6a0d4d1e9c850bdab24d17680f7c8b68bdbef2789a7dd1244ed8"}
Nov 03 23:50:07.588093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.587977    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:07.588093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.587993    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" event={"ID":"211ddd11-9302-4c85-9330-0269815f2af4","Type":"ContainerStarted","Data":"ef92c7e8be7f87880e5214a82da8669f111be120e9293f9b7cdeebf26f806370"}
Nov 03 23:50:07.631166 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.631115    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" podStartSLOduration=1.631098368 podStartE2EDuration="1.631098368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:07.628380711 +0000 UTC m=+1223.519442646" watchObservedRunningTime="2026-11-03 23:50:07.631098368 +0000 UTC m=+1223.522160302"
Nov 03 23:50:07.651925 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:07.651875    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" podStartSLOduration=1.651861098 podStartE2EDuration="1.651861098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:07.650358965 +0000 UTC m=+1223.541420900" watchObservedRunningTime="2026-11-03 23:50:07.651861098 +0000 UTC m=+1223.542923027"
Nov 03 23:50:09.794409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:09.793888    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:50:09.794409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:09.794165    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" podUID="70869874-b7ef-4a34-a97c-820f073f400d" containerName="payload-pre-processing" containerID="cri-o://4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986" gracePeriod=30
Nov 03 23:50:10.434255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.434222    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:50:10.534988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.534896    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\") pod \"70869874-b7ef-4a34-a97c-820f073f400d\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") "
Nov 03 23:50:10.535179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.535103    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\" (UniqueName: \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\") pod \"70869874-b7ef-4a34-a97c-820f073f400d\" (UID: \"70869874-b7ef-4a34-a97c-820f073f400d\") "
Nov 03 23:50:10.583416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.582870    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume" pod "70869874-b7ef-4a34-a97c-820f073f400d" (UID: "70869874-b7ef-4a34-a97c-820f073f400d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:10.612461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.611923    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6" pod "70869874-b7ef-4a34-a97c-820f073f400d" (UID: "70869874-b7ef-4a34-a97c-820f073f400d"). InnerVolumeSpecName "kube-api-access-vgkt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:10.615557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.615522    6430 generic.go:356] "Generic (PLEG): container finished" podID="70869874-b7ef-4a34-a97c-820f073f400d" containerID="4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986" exitCode=0
Nov 03 23:50:10.615700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.615570    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"
Nov 03 23:50:10.615700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.615632    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" event={"ID":"70869874-b7ef-4a34-a97c-820f073f400d","Type":"ContainerDied","Data":"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"}
Nov 03 23:50:10.615817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.615713    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl" event={"ID":"70869874-b7ef-4a34-a97c-820f073f400d","Type":"ContainerDied","Data":"ffe28b9ea752e7d8a559ee8a773ff0d470547f9537b872a0ad39f614efb62e3a"}
Nov 03 23:50:10.615817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.615748    6430 scope.go:122] "RemoveContainer" containerID="4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"
Nov 03 23:50:10.640417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.638001    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70869874-b7ef-4a34-a97c-820f073f400d-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:10.640417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.638034    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkt6\" (UniqueName: \"kubernetes.io/projected/70869874-b7ef-4a34-a97c-820f073f400d-kube-api-access-vgkt6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:10.646612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.646575    6430 scope.go:122] "RemoveContainer" containerID="4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"
Nov 03 23:50:10.647114 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:10.646993    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986\": container with ID starting with 4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986 not found: ID does not exist" containerID="4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"
Nov 03 23:50:10.647114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.647033    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986"} err="failed to get container status \"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986\": rpc error: code = NotFound desc = could not find container \"4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986\": container with ID starting with 4bd36824f38031b039ff191be83d7cc3883c2dd69c3e29c3b3ae12badf3fe986 not found: ID does not exist"
Nov 03 23:50:10.651341 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.651299    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:50:10.657045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:10.657017    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8cfc3c-7cccc547c5-q2ftl"]
Nov 03 23:50:12.339417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:12.314617    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70869874-b7ef-4a34-a97c-820f073f400d" path="/var/lib/kubelet/pods/70869874-b7ef-4a34-a97c-820f073f400d/volumes"
Nov 03 23:50:13.041713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.040621    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"]
Nov 03 23:50:13.052869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.052815    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fcced1e-9dkgh"]
Nov 03 23:50:13.409242 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.409206    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:50:13.413467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.410290    6430 cpu_manager.go:414] "containerMap: removing container" podUID="70869874-b7ef-4a34-a97c-820f073f400d" containerName="payload-pre-processing"
Nov 03 23:50:13.413467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.410320    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70869874-b7ef-4a34-a97c-820f073f400d" containerName="payload-pre-processing"
Nov 03 23:50:13.413467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.410514    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="70869874-b7ef-4a34-a97c-820f073f400d" containerName="payload-pre-processing"
Nov 03 23:50:13.416191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.416170    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.429546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.425791    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-532e80-serving-cert\""
Nov 03 23:50:13.434131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.430495    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:50:13.565767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.565715    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.565933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.565849    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzml\" (UniqueName: \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.565970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.565944    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.566090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.566077    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.672623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.672519    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.676426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.673006    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.683416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.676782    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.683827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.683796    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzml\" (UniqueName: \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.684420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.682207    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.739500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.739465    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzml\" (UniqueName: \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.739980 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.739947    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.740407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.740360    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\") pod \"maas-api-e2e-api-a-532e80-6c65c6b45c-c642f\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:13.765274 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:13.752689    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:14.122411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:14.114098    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:50:14.159430 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:14.153498    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95a1028_5b73_4b08_9b04_d57f8ad646a5.slice/crio-595b0b9bc88be77395558df83984fbe2d0aa6e5bc89b4547c02b491c1f35f967 WatchSource:0}: Error finding container 595b0b9bc88be77395558df83984fbe2d0aa6e5bc89b4547c02b491c1f35f967: Status 404 returned error can't find the container with id 595b0b9bc88be77395558df83984fbe2d0aa6e5bc89b4547c02b491c1f35f967
Nov 03 23:50:14.262501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:14.262325    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c8f878-422a-44aa-bc86-4361e652f713" path="/var/lib/kubelet/pods/22c8f878-422a-44aa-bc86-4361e652f713/volumes"
Nov 03 23:50:14.737359 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:14.737317    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" event={"ID":"e95a1028-5b73-4b08-9b04-d57f8ad646a5","Type":"ContainerStarted","Data":"595b0b9bc88be77395558df83984fbe2d0aa6e5bc89b4547c02b491c1f35f967"}
Nov 03 23:50:15.829409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:15.827193    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" event={"ID":"e95a1028-5b73-4b08-9b04-d57f8ad646a5","Type":"ContainerStarted","Data":"98191d2906ece4a22d1583c71fa674e1085038df0be5efcb32659dbeb8d16472"}
Nov 03 23:50:15.829409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:15.828540    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:15.857609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:15.857543    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" podStartSLOduration=2.342568421 podStartE2EDuration="2.857525695s" totalImagesPullingTime="514.957274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:15.854372726 +0000 UTC m=+1231.745434659" watchObservedRunningTime="2026-11-03 23:50:15.857525695 +0000 UTC m=+1231.748587631"
Nov 03 23:50:16.174458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.173033    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:50:16.182316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.182282    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.192555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.192526    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c20f75-openshift-default-dockercfg-gtssd\""
Nov 03 23:50:16.208070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.195477    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.319626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.319780    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.319862    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.319896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.319944    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.320010    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8wd\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.320045    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.320116    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.320423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.320151    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.425668    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8wd\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426143    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426255    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426296    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426438    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426601    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426707    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426750    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.426804    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.427662    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.431280 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.428307    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.434000 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.433846    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.434490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.434460    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.434733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.434699    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.440771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.440740    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.442468 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.442348    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8wd\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.447961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.447103    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.448986 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.448963    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\") pod \"e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") " pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:16.524990 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:16.524946    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:17.050148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.050115    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:50:17.207948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.188643    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:17.207948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.188755    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:17.207948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.188796    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:17.848763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.848715    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerStarted","Data":"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"}
Nov 03 23:50:17.848972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.848829    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerStarted","Data":"f718729afbf8957e0400da8780dbf73bfadd9b3547a37ed841a84d1e2d7c4602"}
Nov 03 23:50:17.879023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:17.878967    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podStartSLOduration=1.878953463 podStartE2EDuration="1.878953463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:17.876435161 +0000 UTC m=+1233.767497091" watchObservedRunningTime="2026-11-03 23:50:17.878953463 +0000 UTC m=+1233.770015396"
Nov 03 23:50:18.526020 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.525982    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:18.531949 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.531922    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:18.598303 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.598272    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:50:18.598517 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.598340    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:50:18.855752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.855716    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:18.856934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:18.856913    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:50:20.341472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:20.341438    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:50:20.341968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:20.341657    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" containerID="cri-o://994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10" gracePeriod=30
Nov 03 23:50:21.489937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.489901    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:21.495373 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.495351    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.503142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.503117    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c20f75\""
Nov 03 23:50:21.503519 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.503491    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:21.575730 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.575686    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.575955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.575852    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfd6k\" (UniqueName: \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.685433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.680567    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.685433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.680746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfd6k\" (UniqueName: \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.685433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.682273    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.714829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.714764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfd6k\" (UniqueName: \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\") pod \"payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:21.820027 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:21.819994    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:22.302721 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:22.302690    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9708f1_a325_4a59_b767_95be0db3845f.slice/crio-73eb1e9781b0e8770cd27f1c4907cd7422701cdb654604c802de4c618da7bc22 WatchSource:0}: Error finding container 73eb1e9781b0e8770cd27f1c4907cd7422701cdb654604c802de4c618da7bc22: Status 404 returned error can't find the container with id 73eb1e9781b0e8770cd27f1c4907cd7422701cdb654604c802de4c618da7bc22
Nov 03 23:50:22.309276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.309250    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:22.854100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.854067    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:50:22.909830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.909793    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" event={"ID":"6d9708f1-a325-4a59-b767-95be0db3845f","Type":"ContainerStarted","Data":"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"}
Nov 03 23:50:22.910014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.909870    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" event={"ID":"6d9708f1-a325-4a59-b767-95be0db3845f","Type":"ContainerStarted","Data":"73eb1e9781b0e8770cd27f1c4907cd7422701cdb654604c802de4c618da7bc22"}
Nov 03 23:50:22.910132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.910116    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:22.933354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:22.933258    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" podStartSLOduration=1.933240042 podStartE2EDuration="1.933240042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:22.932943912 +0000 UTC m=+1238.824005847" watchObservedRunningTime="2026-11-03 23:50:22.933240042 +0000 UTC m=+1238.824301980"
Nov 03 23:50:23.630513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.630471    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:50:23.630763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.630738    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy" containerID="cri-o://299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956" gracePeriod=30
Nov 03 23:50:23.635198 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.635170    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:50:23.635809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.635521    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy" containerID="cri-o://b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f" gracePeriod=30
Nov 03 23:50:23.639272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.639248    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:50:23.639502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:23.639477    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy" containerID="cri-o://093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a" gracePeriod=30
Nov 03 23:50:24.300059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:24.300023    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:50:24.300464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:24.300083    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:50:25.633414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.633327    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:50:25.785358 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785324    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785358 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785359    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785464    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785581    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785615    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785703    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.785879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785771    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:25.785879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785801    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:25.785879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785855    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:25.786092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785809    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\") pod \"9042c877-e246-4e1c-90ea-81200d19e31e\" (UID: \"9042c877-e246-4e1c-90ea-81200d19e31e\") "
Nov 03 23:50:25.786092 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.785918    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:25.786188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786130    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:25.786674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786649    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.786674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786670    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.786838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786680    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.786838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786691    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.786838 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.786742    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9042c877-e246-4e1c-90ea-81200d19e31e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.787945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.787917    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:25.788035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.787966    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "kube-api-access-d8r2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:25.788144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.788121    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:25.788240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.788223    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo" pod "9042c877-e246-4e1c-90ea-81200d19e31e" (UID: "9042c877-e246-4e1c-90ea-81200d19e31e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:25.888319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.888225    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8r2p\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-kube-api-access-d8r2p\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.888319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.888264    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9042c877-e246-4e1c-90ea-81200d19e31e-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.888319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.888274    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9042c877-e246-4e1c-90ea-81200d19e31e-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:25.888319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:25.888284    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9042c877-e246-4e1c-90ea-81200d19e31e-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:26.013416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.012515    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"
Nov 03 23:50:26.032442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.030552    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerDied","Data":"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"}
Nov 03 23:50:26.032442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.030685    6430 scope.go:122] "RemoveContainer" containerID="994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"
Nov 03 23:50:26.032442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.012267    6430 generic.go:356] "Generic (PLEG): container finished" podID="9042c877-e246-4e1c-90ea-81200d19e31e" containerID="994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10" exitCode=0
Nov 03 23:50:26.032442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.030942    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp" event={"ID":"9042c877-e246-4e1c-90ea-81200d19e31e","Type":"ContainerDied","Data":"f7b4deb73b4b34587bf8947950b3639006e35359a1244be4604e27f0539ad6d4"}
Nov 03 23:50:26.080675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.072576    6430 scope.go:122] "RemoveContainer" containerID="736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.105335    6430 scope.go:122] "RemoveContainer" containerID="994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:26.105743    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10\": container with ID starting with 994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10 not found: ID does not exist" containerID="994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.105780    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10"} err="failed to get container status \"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10\": rpc error: code = NotFound desc = could not find container \"994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10\": container with ID starting with 994a5a4f72d73394876bb8ce2caa446aa65fe70c009a0323934da3b6c0fe0a10 not found: ID does not exist"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.105804    6430 scope.go:122] "RemoveContainer" containerID="736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:26.106118    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a\": container with ID starting with 736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a not found: ID does not exist" containerID="736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"
Nov 03 23:50:26.108569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.106150    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a"} err="failed to get container status \"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a\": rpc error: code = NotFound desc = could not find container \"736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a\": container with ID starting with 736ecde50ffe38760dbd7dde81d159d3008f497a3abd9b8e7470e995a607069a not found: ID does not exist"
Nov 03 23:50:26.135420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.131462    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:50:26.135420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.132173    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8cfc3c-openshift-default-fbb9778f7-8qdcp"]
Nov 03 23:50:26.391294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:26.391250    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" path="/var/lib/kubelet/pods/9042c877-e246-4e1c-90ea-81200d19e31e/volumes"
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.126272    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.127259    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.127277    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.127340    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.127348    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.128905 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.127573    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.136434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.136326    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.142212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.142020    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:50:27.151955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.151744    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-695f03\""
Nov 03 23:50:27.162259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.162237    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:50:27.163694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.163670    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9042c877-e246-4e1c-90ea-81200d19e31e" containerName="istio-proxy"
Nov 03 23:50:27.167804 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.167785    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.173806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.173743    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-695f03-dockercfg-fcgvq\""
Nov 03 23:50:27.180017 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.179992    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:50:27.330412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.320894    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8hr\" (UniqueName: \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.330412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.321013    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzww7\" (UniqueName: \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.330412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.321307    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.330412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.321368    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.425463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.425063    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.425463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.425146    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.429410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.426252    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.429410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.426490    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8hr\" (UniqueName: \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.429410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.427678    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzww7\" (UniqueName: \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.429410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.427225    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.458411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.455374    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzww7\" (UniqueName: \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\") pod \"payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.467616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.464366    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8hr\" (UniqueName: \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\") pod \"payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.467616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.467071    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:27.507324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.506136    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:27.904757 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.904729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:50:27.905490 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:27.905462    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode129e2dc_6051_45ee_9c70_ef61e9283038.slice/crio-901bdfbb44bb9a6368af616e8b24708aa8270cecc9e8e7c5f393eb78c020c4ea WatchSource:0}: Error finding container 901bdfbb44bb9a6368af616e8b24708aa8270cecc9e8e7c5f393eb78c020c4ea: Status 404 returned error can't find the container with id 901bdfbb44bb9a6368af616e8b24708aa8270cecc9e8e7c5f393eb78c020c4ea
Nov 03 23:50:27.915228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:27.915205    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:50:27.918540 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:27.918519    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6a7cc_7654_4aa7_b731_d6a1977340a8.slice/crio-f5d3df9a6306e2fe7ed28f2fd29f66a90a6139cb02a4958d0553983c66c954ae WatchSource:0}: Error finding container f5d3df9a6306e2fe7ed28f2fd29f66a90a6139cb02a4958d0553983c66c954ae: Status 404 returned error can't find the container with id f5d3df9a6306e2fe7ed28f2fd29f66a90a6139cb02a4958d0553983c66c954ae
Nov 03 23:50:28.202835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.202791    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" event={"ID":"e129e2dc-6051-45ee-9c70-ef61e9283038","Type":"ContainerStarted","Data":"fbd3a782c31f194e8f433d2622e9cc2f1090b2f108eeec02cc6f30b458e2f497"}
Nov 03 23:50:28.203288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.202892    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" event={"ID":"e129e2dc-6051-45ee-9c70-ef61e9283038","Type":"ContainerStarted","Data":"901bdfbb44bb9a6368af616e8b24708aa8270cecc9e8e7c5f393eb78c020c4ea"}
Nov 03 23:50:28.203288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.202926    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:28.204910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.204886    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" event={"ID":"eda6a7cc-7654-4aa7-b731-d6a1977340a8","Type":"ContainerStarted","Data":"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"}
Nov 03 23:50:28.204996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.204928    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" event={"ID":"eda6a7cc-7654-4aa7-b731-d6a1977340a8","Type":"ContainerStarted","Data":"f5d3df9a6306e2fe7ed28f2fd29f66a90a6139cb02a4958d0553983c66c954ae"}
Nov 03 23:50:28.205183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.205164    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:28.225424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.225313    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" podStartSLOduration=1.225301108 podStartE2EDuration="1.225301108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:28.222631948 +0000 UTC m=+1244.113693882" watchObservedRunningTime="2026-11-03 23:50:28.225301108 +0000 UTC m=+1244.116363041"
Nov 03 23:50:28.246483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.246436    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" podStartSLOduration=1.246422223 podStartE2EDuration="1.246422223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:28.245308419 +0000 UTC m=+1244.136370354" watchObservedRunningTime="2026-11-03 23:50:28.246422223 +0000 UTC m=+1244.137484156"
Nov 03 23:50:28.921159 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.921084    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:50:28.994601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.994573    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.994768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.994697    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.994847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.994783    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.994951    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995036    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995063    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995089    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995153    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995177    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995257    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995277    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995291    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\") pod \"cb070961-7e25-4bb4-ae8b-1953dfffa534\" (UID: \"cb070961-7e25-4bb4-ae8b-1953dfffa534\") "
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.995832    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.996479    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.996604    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.996624    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.996641    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.996666    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb070961-7e25-4bb4-ae8b-1953dfffa534-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.997179    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "kube-api-access-rp6z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.997796    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:28.998072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.997839    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:28.998845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:28.998495    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token" pod "cb070961-7e25-4bb4-ae8b-1953dfffa534" (UID: "cb070961-7e25-4bb4-ae8b-1953dfffa534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:29.000626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.000606    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:50:29.097911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.097876    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.097934    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.097970    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098045    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098077    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098150    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098174    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098215    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098242    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098303    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098369    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098444    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098478    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098505    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\") pod \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\" (UID: \"c3b487d7-f998-4f38-bbe4-54aba5fccfc1\") "
Nov 03 23:50:29.098590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098564    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098596    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098648    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098686    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\") pod \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\" (UID: \"724b281b-1a95-47fd-a1c8-acec7a88d2f7\") "
Nov 03 23:50:29.098830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.098724    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:29.099321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099293    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.099484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099354    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.099484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099374    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.099484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099410    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.099662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:29.099899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099877    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.099993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.099875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100052    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100056    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100091    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100503    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100532    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.100824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100760    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6z7\" (UniqueName: \"kubernetes.io/projected/cb070961-7e25-4bb4-ae8b-1953dfffa534-kube-api-access-rp6z7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.100824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100789    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.100824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.100804    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb070961-7e25-4bb4-ae8b-1953dfffa534-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.101126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.101096    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:29.102649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.102620    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "kube-api-access-fjtbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:29.103088 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.103034    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.103293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.103068    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:29.103403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.103297    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:29.103499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.103369    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "kube-api-access-76djb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:29.104518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.104493    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token" pod "c3b487d7-f998-4f38-bbe4-54aba5fccfc1" (UID: "c3b487d7-f998-4f38-bbe4-54aba5fccfc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:29.104773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.104754    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo" pod "724b281b-1a95-47fd-a1c8-acec7a88d2f7" (UID: "724b281b-1a95-47fd-a1c8-acec7a88d2f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.201939    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.201970    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.201979    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.201990    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.201999    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202007    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202015    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202023    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202031    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtbv\" (UniqueName: \"kubernetes.io/projected/724b281b-1a95-47fd-a1c8-acec7a88d2f7-kube-api-access-fjtbv\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202039    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202048    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202055    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202066    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76djb\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-kube-api-access-76djb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202075    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202083    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202090    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724b281b-1a95-47fd-a1c8-acec7a88d2f7-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.202402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.202099    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b487d7-f998-4f38-bbe4-54aba5fccfc1-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:29.213033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.212998    6430 generic.go:356] "Generic (PLEG): container finished" podID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerID="093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a" exitCode=0
Nov 03 23:50:29.213512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.213043    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"
Nov 03 23:50:29.213512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.213080    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" event={"ID":"cb070961-7e25-4bb4-ae8b-1953dfffa534","Type":"ContainerDied","Data":"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"}
Nov 03 23:50:29.213512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.213136    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk" event={"ID":"cb070961-7e25-4bb4-ae8b-1953dfffa534","Type":"ContainerDied","Data":"4fec17a6950e2f2a06c0e994c7d986eed6b25f39a543841b5a431f4f205d9c25"}
Nov 03 23:50:29.213512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.213161    6430 scope.go:122] "RemoveContainer" containerID="093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"
Nov 03 23:50:29.215538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.215517    6430 generic.go:356] "Generic (PLEG): container finished" podID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerID="b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f" exitCode=0
Nov 03 23:50:29.215642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.215580    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"
Nov 03 23:50:29.215642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.215598    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" event={"ID":"724b281b-1a95-47fd-a1c8-acec7a88d2f7","Type":"ContainerDied","Data":"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"}
Nov 03 23:50:29.215779 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.215652    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4" event={"ID":"724b281b-1a95-47fd-a1c8-acec7a88d2f7","Type":"ContainerDied","Data":"a1c4f84914afd7bcfa255d2f247038c9ddfab8b7006eeb86db0fb2d1e82fd710"}
Nov 03 23:50:29.217106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.217085    6430 generic.go:356] "Generic (PLEG): container finished" podID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerID="299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956" exitCode=0
Nov 03 23:50:29.217206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.217122    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"
Nov 03 23:50:29.217206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.217164    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" event={"ID":"c3b487d7-f998-4f38-bbe4-54aba5fccfc1","Type":"ContainerDied","Data":"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"}
Nov 03 23:50:29.217206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.217203    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp" event={"ID":"c3b487d7-f998-4f38-bbe4-54aba5fccfc1","Type":"ContainerDied","Data":"3dabd35f8f25d567109d0c19e55f7ec38a09058292d15632945ef8e7d21859df"}
Nov 03 23:50:29.225149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.225101    6430 scope.go:122] "RemoveContainer" containerID="093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"
Nov 03 23:50:29.225454 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:29.225424    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a\": container with ID starting with 093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a not found: ID does not exist" containerID="093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"
Nov 03 23:50:29.225557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.225464    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a"} err="failed to get container status \"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a\": rpc error: code = NotFound desc = could not find container \"093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a\": container with ID starting with 093776e1845cc37568f9cdfe3f97d6b2b0b53e9979ecf6e3080a215139faee3a not found: ID does not exist"
Nov 03 23:50:29.225557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.225487    6430 scope.go:122] "RemoveContainer" containerID="b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"
Nov 03 23:50:29.237944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.237928    6430 scope.go:122] "RemoveContainer" containerID="b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"
Nov 03 23:50:29.238309 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:29.238234    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f\": container with ID starting with b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f not found: ID does not exist" containerID="b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"
Nov 03 23:50:29.238309 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.238269    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f"} err="failed to get container status \"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f\": rpc error: code = NotFound desc = could not find container \"b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f\": container with ID starting with b1830451af7e519708ae982d1ca146aa07ceee4f012789ac6cbcc0ac7205c84f not found: ID does not exist"
Nov 03 23:50:29.238309 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.238306    6430 scope.go:122] "RemoveContainer" containerID="299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"
Nov 03 23:50:29.254730 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.254706    6430 scope.go:122] "RemoveContainer" containerID="299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"
Nov 03 23:50:29.256034 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:29.256006    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956\": container with ID starting with 299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956 not found: ID does not exist" containerID="299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"
Nov 03 23:50:29.256145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.256044    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956"} err="failed to get container status \"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956\": rpc error: code = NotFound desc = could not find container \"299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956\": container with ID starting with 299a1569c9d0aa347c70b1f1e1568205dc2463d7b52abdbf8b5183273613d956 not found: ID does not exist"
Nov 03 23:50:29.262815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.262789    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:50:29.267586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.267564    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-9n5wp"]
Nov 03 23:50:29.275527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.275506    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:50:29.292185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.292155    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-f67v4"]
Nov 03 23:50:29.306194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.306168    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:50:29.310847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:29.310820    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-4qgqk"]
Nov 03 23:50:30.251683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:30.251644    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" path="/var/lib/kubelet/pods/724b281b-1a95-47fd-a1c8-acec7a88d2f7/volumes"
Nov 03 23:50:30.252360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:30.252342    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" path="/var/lib/kubelet/pods/c3b487d7-f998-4f38-bbe4-54aba5fccfc1/volumes"
Nov 03 23:50:30.252992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:30.252972    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" path="/var/lib/kubelet/pods/cb070961-7e25-4bb4-ae8b-1953dfffa534/volumes"
Nov 03 23:50:31.219566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.219513    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:50:31.222515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.222487    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy"
Nov 03 23:50:31.222706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.222691    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy"
Nov 03 23:50:31.222816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.222806    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy"
Nov 03 23:50:31.222896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.222886    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy"
Nov 03 23:50:31.223006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.222994    6430 cpu_manager.go:414] "containerMap: removing container" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy"
Nov 03 23:50:31.223080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.223071    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy"
Nov 03 23:50:31.223352 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.223338    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb070961-7e25-4bb4-ae8b-1953dfffa534" containerName="istio-proxy"
Nov 03 23:50:31.223509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.223497    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="724b281b-1a95-47fd-a1c8-acec7a88d2f7" containerName="istio-proxy"
Nov 03 23:50:31.223602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.223592    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b487d7-f998-4f38-bbe4-54aba5fccfc1" containerName="istio-proxy"
Nov 03 23:50:31.274428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.272780    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:50:31.274428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.272893    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.281145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.281121    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c83d52-openshift-default-dockercfg-86k8x\""
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.419884    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420143    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2qn\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420209    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420260    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420288    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420411    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420537    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.427419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.420626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.526134 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.526036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.526897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.526869    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2qn\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527076    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527216    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527307    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527453    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527681    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.527874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.527854    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.528093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.528076    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.528690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.528670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.528817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.526757    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.529992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.529966    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.530339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.530314    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.540151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.531094    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.552983 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.552952    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.554955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.554926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.562060 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.561963    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2qn\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.573157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.573077    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\") pod \"e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") " pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.595416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.590013    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:31.979721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.979681    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:50:31.987366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.987336    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:31.994827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.994792    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:50:31.995058 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:31.995039    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c20f75-serving-cert\""
Nov 03 23:50:32.055994 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.055947    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.055994 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.055994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjw4\" (UniqueName: \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.056266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.056170    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.056266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.056241    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.097638 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.097291    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:50:32.100779 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:32.100735    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc445bf6_97e7_46d7_9ae4_16a064f6aa4f.slice/crio-73401ef3ea16ca08b3e9756d9deb94b23c0309d7192d492dc866c7945ca3ac13 WatchSource:0}: Error finding container 73401ef3ea16ca08b3e9756d9deb94b23c0309d7192d492dc866c7945ca3ac13: Status 404 returned error can't find the container with id 73401ef3ea16ca08b3e9756d9deb94b23c0309d7192d492dc866c7945ca3ac13
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.159803    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.159909    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.159954    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.161769    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.161819    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjw4\" (UniqueName: \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.161940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.162413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.162033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.190444 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:32.178836    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-serving-cert: secret "maas-api-e2e-ipp-a-c20f75-serving-cert" not found
Nov 03 23:50:32.190444 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:32.178924    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls podName:f8115980-ca51-4c61-9187-4f2f646e2e09 nodeName:}" failed. No retries permitted until 2026-11-03 23:50:32.678901753 +0000 UTC m=+1248.569963679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls") pod "maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" (UID: "f8115980-ca51-4c61-9187-4f2f646e2e09") : secret "maas-api-e2e-ipp-a-c20f75-serving-cert" not found
Nov 03 23:50:32.190444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.179726    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.199817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.192038    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.236424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.234996    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjw4\" (UniqueName: \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.305358 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.305325    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerStarted","Data":"73401ef3ea16ca08b3e9756d9deb94b23c0309d7192d492dc866c7945ca3ac13"}
Nov 03 23:50:32.695775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.695674    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.701934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.701899    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:32.937798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:32.937758    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:33.403416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.366534    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerStarted","Data":"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"}
Nov 03 23:50:33.486238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.486181    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podStartSLOduration=2.486150709 podStartE2EDuration="2.486150709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:33.479722823 +0000 UTC m=+1249.370784758" watchObservedRunningTime="2026-11-03 23:50:33.486150709 +0000 UTC m=+1249.377212642"
Nov 03 23:50:33.660503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.658644    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:33.660503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.658786    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:33.757270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.757152    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:50:33.761628 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:33.760559    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8115980_ca51_4c61_9187_4f2f646e2e09.slice/crio-fb24275d75c336aa11154dc788949d68e001d9ccae4f63ba8ce258108382ee5e WatchSource:0}: Error finding container fb24275d75c336aa11154dc788949d68e001d9ccae4f63ba8ce258108382ee5e: Status 404 returned error can't find the container with id fb24275d75c336aa11154dc788949d68e001d9ccae4f63ba8ce258108382ee5e
Nov 03 23:50:33.762632 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.762613    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:50:33.934841 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:33.934754    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:34.389382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:34.389350    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" event={"ID":"f8115980-ca51-4c61-9187-4f2f646e2e09","Type":"ContainerStarted","Data":"fb24275d75c336aa11154dc788949d68e001d9ccae4f63ba8ce258108382ee5e"}
Nov 03 23:50:34.389856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:34.389816    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:34.390534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:34.390511    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:50:34.859009 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:34.858954    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:50:35.095462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:35.095408    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:50:35.095687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:35.095657    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy" containerID="cri-o://b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543" gracePeriod=30
Nov 03 23:50:35.398085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:35.398044    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" event={"ID":"f8115980-ca51-4c61-9187-4f2f646e2e09","Type":"ContainerStarted","Data":"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"}
Nov 03 23:50:35.398275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:35.398118    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:35.429410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:35.421824    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" podStartSLOduration=3.842203988 podStartE2EDuration="4.421805047s" totalImagesPullingTime="579.601059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:35.421659922 +0000 UTC m=+1251.312721857" watchObservedRunningTime="2026-11-03 23:50:35.421805047 +0000 UTC m=+1251.312866982"
Nov 03 23:50:37.038626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.038590    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:50:37.043091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.043068    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.051716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.051668    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c83d52\""
Nov 03 23:50:37.052129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.051780    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c83d52-dockercfg-hs67t\""
Nov 03 23:50:37.054232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.054213    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:50:37.090531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.090475    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:50:37.095634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.095608    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.105351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.105323    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:50:37.208411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.176209    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnmd\" (UniqueName: \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.212408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.208835    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.218669 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.218626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.218877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.218860    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg89z\" (UniqueName: \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.360416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.358734    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.360416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.358893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.360416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.358939    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg89z\" (UniqueName: \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.360416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.359106    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnmd\" (UniqueName: \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.373038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.372598    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.386719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.386657    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.418547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.418513    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnmd\" (UniqueName: \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\") pod \"payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.419075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.419049    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:37.428487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.427373    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg89z\" (UniqueName: \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\") pod \"payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.672252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.671692    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:37.811732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.811697    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:50:37.987354 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:37.987307    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9361512_357a_47b6_97e9_199e67f673af.slice/crio-8f1eaf11b407b284f6fc542fca302e8bd56670bee4e18f3f47f8771af84bbc05 WatchSource:0}: Error finding container 8f1eaf11b407b284f6fc542fca302e8bd56670bee4e18f3f47f8771af84bbc05: Status 404 returned error can't find the container with id 8f1eaf11b407b284f6fc542fca302e8bd56670bee4e18f3f47f8771af84bbc05
Nov 03 23:50:37.987585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:37.987564    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:50:38.178576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.178536    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:50:38.202634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.202590    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.216408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.205580    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:50:38.254469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.254379    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-695f03-serving-cert\""
Nov 03 23:50:38.412073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.412029    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.412263 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.412159    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.412356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.412319    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbv6\" (UniqueName: \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.412442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.412421    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.490194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.490155    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" event={"ID":"cfcfa3a3-89bf-4fef-8969-69046c44a1d4","Type":"ContainerStarted","Data":"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"}
Nov 03 23:50:38.490436 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.490234    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" event={"ID":"cfcfa3a3-89bf-4fef-8969-69046c44a1d4","Type":"ContainerStarted","Data":"77c97cbd40e629acb56d47ec478fd4397cb4dd97dfd5c59bea2c242e330ca1b2"}
Nov 03 23:50:38.490501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.490463    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:38.492326 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.492274    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" event={"ID":"c9361512-357a-47b6-97e9-199e67f673af","Type":"ContainerStarted","Data":"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"}
Nov 03 23:50:38.492477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.492345    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" event={"ID":"c9361512-357a-47b6-97e9-199e67f673af","Type":"ContainerStarted","Data":"8f1eaf11b407b284f6fc542fca302e8bd56670bee4e18f3f47f8771af84bbc05"}
Nov 03 23:50:38.492477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.492373    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:38.514145 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.514029    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" podStartSLOduration=1.514011031 podStartE2EDuration="1.514011031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:38.510206373 +0000 UTC m=+1254.401268307" watchObservedRunningTime="2026-11-03 23:50:38.514011031 +0000 UTC m=+1254.405072965"
Nov 03 23:50:38.514897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.514870    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.514981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.514926    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.515047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.515002    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbv6\" (UniqueName: \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.515105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.515050    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.515918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.515891    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.518682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.518377    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.518873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.518849    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.530362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.530329    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbv6\" (UniqueName: \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\") pod \"maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.536488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.536430    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" podStartSLOduration=1.536375541 podStartE2EDuration="1.536375541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:38.531586224 +0000 UTC m=+1254.422648160" watchObservedRunningTime="2026-11-03 23:50:38.536375541 +0000 UTC m=+1254.427437478"
Nov 03 23:50:38.538837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.538809    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:38.930408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:38.913899    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:50:39.221409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:39.221310    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:50:39.221795 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:39.221381    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:50:39.593211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:39.585960    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" event={"ID":"f025ba1a-bdfa-4812-8e0c-1c7895147dc5","Type":"ContainerStarted","Data":"286b41d4ef3a8c46e5ac64b103763a280525ff427699ebe620d663b348f6e317"}
Nov 03 23:50:40.469592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.469548    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:50:40.573663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573763    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573798    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573826    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573854    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573928    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.573992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.573943    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.574154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574007    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.574154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574046    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.574154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574101    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\") pod \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\" (UID: \"38d70def-e6b7-4ed1-a7cf-0f6413475b23\") "
Nov 03 23:50:40.574297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574196    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.574602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574526    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.574602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574552    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.574770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:40.574881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574861    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.574881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574885    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.574997 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574899    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.574997 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574912    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.574997 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.574926    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.576272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.576249    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:50:40.576557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.576527    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:50:40.576689 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.576627    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "kube-api-access-qgw86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:40.576689 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.576675    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token" pod "38d70def-e6b7-4ed1-a7cf-0f6413475b23" (UID: "38d70def-e6b7-4ed1-a7cf-0f6413475b23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:40.595902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.595859    6430 generic.go:356] "Generic (PLEG): container finished" podID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerID="b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543" exitCode=0
Nov 03 23:50:40.596047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.595907    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"
Nov 03 23:50:40.596047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.595936    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" event={"ID":"38d70def-e6b7-4ed1-a7cf-0f6413475b23","Type":"ContainerDied","Data":"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"}
Nov 03 23:50:40.596047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.596040    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff" event={"ID":"38d70def-e6b7-4ed1-a7cf-0f6413475b23","Type":"ContainerDied","Data":"ddd4db1edbb6dd087a41ba7e89db35296626df41303ddec76f3d31b00d23d4e5"}
Nov 03 23:50:40.596210 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.596069    6430 scope.go:122] "RemoveContainer" containerID="b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"
Nov 03 23:50:40.598643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.598413    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" event={"ID":"f025ba1a-bdfa-4812-8e0c-1c7895147dc5","Type":"ContainerStarted","Data":"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"}
Nov 03 23:50:40.598643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.598607    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:40.624289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.621053    6430 scope.go:122] "RemoveContainer" containerID="b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"
Nov 03 23:50:40.624289 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:40.621570    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543\": container with ID starting with b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543 not found: ID does not exist" containerID="b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"
Nov 03 23:50:40.624289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.621606    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543"} err="failed to get container status \"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543\": rpc error: code = NotFound desc = could not find container \"b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543\": container with ID starting with b6f9177b9f4f3e6dc9899a5e3872bfdb638f4c1c78806708266a2fd6f5de7543 not found: ID does not exist"
Nov 03 23:50:40.625589 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.625540    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" podStartSLOduration=2.117938606 podStartE2EDuration="2.625515862s" totalImagesPullingTime="507.577256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:40.622898886 +0000 UTC m=+1256.513960839" watchObservedRunningTime="2026-11-03 23:50:40.625515862 +0000 UTC m=+1256.516577807"
Nov 03 23:50:40.653453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.649584    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:50:40.654584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.654537    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-mcdff"]
Nov 03 23:50:40.676403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.676309    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.676403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.676343    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgw86\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-kube-api-access-qgw86\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.676403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.676357    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:40.676403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:40.676368    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d70def-e6b7-4ed1-a7cf-0f6413475b23-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:41.444165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.444133    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:50:41.609098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.609074    6430 generic.go:356] "Generic (PLEG): container finished" podID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerID="29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e" exitCode=0
Nov 03 23:50:41.609481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.609151    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerDied","Data":"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"}
Nov 03 23:50:41.609714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.609694    6430 scope.go:122] "RemoveContainer" containerID="29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"
Nov 03 23:50:41.611054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.611029    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:41.611137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.611092    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:41.611137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:41.611122    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:42.287417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:42.281132    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" path="/var/lib/kubelet/pods/38d70def-e6b7-4ed1-a7cf-0f6413475b23/volumes"
Nov 03 23:50:42.639181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:42.639142    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerStarted","Data":"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"}
Nov 03 23:50:43.005004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.003665    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:43.005004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.003803    6430 patch_prober.go:29] interesting pod/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:50:43.005004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.003863    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 23:50:43.799503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.796664    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:50:43.799503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.797954    6430 cpu_manager.go:414] "containerMap: removing container" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy"
Nov 03 23:50:43.799503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.797978    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy"
Nov 03 23:50:43.799503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.798223    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d70def-e6b7-4ed1-a7cf-0f6413475b23" containerName="istio-proxy"
Nov 03 23:50:43.805433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.805402    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.834489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.832983    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:50:43.841318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.841282    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c7e425\"/\"kube-root-ca.crt\""
Nov 03 23:50:43.841318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.841302    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c7e425\"/\"praxis-route-c7e425-kserve-self-signed-certs\""
Nov 03 23:50:43.841534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.841329    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c7e425\"/\"default-dockercfg-hrkx9\""
Nov 03 23:50:43.863230 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863198    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.863382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863331    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.863483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863406    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.863524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863484    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hkgf\" (UniqueName: \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.863524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863512    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.863626 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.863541    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.931420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.930693    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c7e425\"/\"openshift-service-ca.crt\""
Nov 03 23:50:43.969489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969378    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.969489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969472    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkgf\" (UniqueName: \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.969746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969510    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.969746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969555    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.969866 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.969931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.969881    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:43.987475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.970359    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.000438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.989768    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.000438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:43.990090    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.004426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.002404    6430 patch_prober.go:29] interesting pod/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:50:44.004426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.002476    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 23:50:44.027421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.018663    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkgf\" (UniqueName: \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.027421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.019343    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.027421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.026753    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\") pod \"praxis-route-c7e425-kserve-77786c6b45-q7r6f\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") " pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.174458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.171043    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:44.536133 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.536098    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:50:44.537801 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:44.537767    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10381fc2_7f4d_4da6_abdb_20cf72960130.slice/crio-3884adebb2e89cf0b9cff367c581eae974c88ef144fd3e291eee3130e0e8f64d WatchSource:0}: Error finding container 3884adebb2e89cf0b9cff367c581eae974c88ef144fd3e291eee3130e0e8f64d: Status 404 returned error can't find the container with id 3884adebb2e89cf0b9cff367c581eae974c88ef144fd3e291eee3130e0e8f64d
Nov 03 23:50:44.721898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:44.721807    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" event={"ID":"10381fc2-7f4d-4da6-abdb-20cf72960130","Type":"ContainerStarted","Data":"3884adebb2e89cf0b9cff367c581eae974c88ef144fd3e291eee3130e0e8f64d"}
Nov 03 23:50:45.002035 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:45.002002    6430 patch_prober.go:29] interesting pod/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:50:45.002340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:45.002064    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 23:50:45.768961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:45.768919    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" event={"ID":"10381fc2-7f4d-4da6-abdb-20cf72960130","Type":"ContainerStarted","Data":"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"}
Nov 03 23:50:45.794782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:45.794722    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" podStartSLOduration=2.794702939 podStartE2EDuration="2.794702939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:45.79069978 +0000 UTC m=+1261.681761717" watchObservedRunningTime="2026-11-03 23:50:45.794702939 +0000 UTC m=+1261.685764876"
Nov 03 23:50:46.004419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:46.003828    6430 patch_prober.go:29] interesting pod/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:50:46.004419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:46.003897    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 23:50:46.172291 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:46.172251    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:46.638055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:46.638016    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:50:47.002048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:47.001973    6430 patch_prober.go:29] interesting pod/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 23:50:47.002048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:47.002028    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 23:50:48.029789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:48.029746    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:48.030309 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:48.030261    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:48.031606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:48.031581    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:50:49.589304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:49.589269    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:50:49.589740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:49.589553    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:50:50.577310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:50.577269    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:51.717556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:51.717520    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:50:54.171553 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:54.171446    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:54.185023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:54.184998    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:55.088320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.088288    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:50:55.577717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.577666    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" podUID="6d9708f1-a325-4a59-b767-95be0db3845f" containerName="payload-processing" containerID="cri-o://292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755" gracePeriod=25
Nov 03 23:50:55.906120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.906093    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:55.992745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.992711    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\") pod \"6d9708f1-a325-4a59-b767-95be0db3845f\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") "
Nov 03 23:50:55.992947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.992880    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\" (UniqueName: \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\") pod \"6d9708f1-a325-4a59-b767-95be0db3845f\" (UID: \"6d9708f1-a325-4a59-b767-95be0db3845f\") "
Nov 03 23:50:55.993193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.993162    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume" pod "6d9708f1-a325-4a59-b767-95be0db3845f" (UID: "6d9708f1-a325-4a59-b767-95be0db3845f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:50:55.993537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.993511    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9708f1-a325-4a59-b767-95be0db3845f-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:55.995184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:55.995152    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k" pod "6d9708f1-a325-4a59-b767-95be0db3845f" (UID: "6d9708f1-a325-4a59-b767-95be0db3845f"). InnerVolumeSpecName "kube-api-access-lfd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:50:56.085722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.085687    6430 generic.go:356] "Generic (PLEG): container finished" podID="6d9708f1-a325-4a59-b767-95be0db3845f" containerID="292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755" exitCode=0
Nov 03 23:50:56.085889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.085759    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"
Nov 03 23:50:56.085889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.085767    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" event={"ID":"6d9708f1-a325-4a59-b767-95be0db3845f","Type":"ContainerDied","Data":"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"}
Nov 03 23:50:56.085889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.085829    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j" event={"ID":"6d9708f1-a325-4a59-b767-95be0db3845f","Type":"ContainerDied","Data":"73eb1e9781b0e8770cd27f1c4907cd7422701cdb654604c802de4c618da7bc22"}
Nov 03 23:50:56.085889 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.085852    6430 scope.go:122] "RemoveContainer" containerID="292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"
Nov 03 23:50:56.095054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.095034    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfd6k\" (UniqueName: \"kubernetes.io/projected/6d9708f1-a325-4a59-b767-95be0db3845f-kube-api-access-lfd6k\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:50:56.097231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.097211    6430 scope.go:122] "RemoveContainer" containerID="292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"
Nov 03 23:50:56.097512 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:50:56.097494    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755\": container with ID starting with 292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755 not found: ID does not exist" containerID="292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"
Nov 03 23:50:56.097566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.097521    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755"} err="failed to get container status \"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755\": rpc error: code = NotFound desc = could not find container \"292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755\": container with ID starting with 292be17a805cfdc75458ac095f9fb6754591196539b1952e44ec1e4c7e8d2755 not found: ID does not exist"
Nov 03 23:50:56.117206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.117177    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:56.121940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.121886    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c20f75-9856848d6-tl65j"]
Nov 03 23:50:56.295821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:56.295775    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9708f1-a325-4a59-b767-95be0db3845f" path="/var/lib/kubelet/pods/6d9708f1-a325-4a59-b767-95be0db3845f/volumes"
Nov 03 23:50:57.403167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.403108    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:50:57.404091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.404068    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6d9708f1-a325-4a59-b767-95be0db3845f" containerName="payload-processing"
Nov 03 23:50:57.404091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.404090    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9708f1-a325-4a59-b767-95be0db3845f" containerName="payload-processing"
Nov 03 23:50:57.404271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.404229    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9708f1-a325-4a59-b767-95be0db3845f" containerName="payload-processing"
Nov 03 23:50:57.410449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.410431    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.416272 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.416251    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-beffe3-openshift-default-dockercfg-mkjgl\""
Nov 03 23:50:57.421477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.421447    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.548533    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.548655    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.548724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.548767    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.548844    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk74v\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.549036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.549084    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.549138    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.552417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.549222    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.653490    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.653659    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.653813    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.654730    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.654903    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.655056    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.655188    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.655263    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.655300    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.655337    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.657472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.656016    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.669420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.666028    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.683480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.678719    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.683480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.680941    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.683480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.681018    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk74v\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.694416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.692062    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.776406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.775456    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:57.820099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:57.819558    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk74v\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\") pod \"e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") " pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:58.033411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:58.033316    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:50:58.331248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:58.331219    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:50:58.331476 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:50:58.331434    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaf11d0_a759_4d46_b4fe_ff126a7e9a82.slice/crio-f669ad2bec5f159c312bee8955251603f5918316d19d3eff7a7408ac807f04c8 WatchSource:0}: Error finding container f669ad2bec5f159c312bee8955251603f5918316d19d3eff7a7408ac807f04c8: Status 404 returned error can't find the container with id f669ad2bec5f159c312bee8955251603f5918316d19d3eff7a7408ac807f04c8
Nov 03 23:50:58.334217 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:58.334187    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:58.334276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:58.334253    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:58.334311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:58.334282    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:50:59.188412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:59.185994    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerStarted","Data":"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"}
Nov 03 23:50:59.188412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:59.186075    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerStarted","Data":"f669ad2bec5f159c312bee8955251603f5918316d19d3eff7a7408ac807f04c8"}
Nov 03 23:50:59.305446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:50:59.303517    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podStartSLOduration=2.303501922 podStartE2EDuration="2.303501922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:59.292370273 +0000 UTC m=+1275.183432209" watchObservedRunningTime="2026-11-03 23:50:59.303501922 +0000 UTC m=+1275.194563857"
Nov 03 23:51:00.033742 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:00.033702    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:00.040506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:00.040475    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:00.193796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:00.193762    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:00.194783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:00.194759    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:03.542413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:03.531612    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:51:03.542413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:03.531899    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy" containerID="cri-o://5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1" gracePeriod=30
Nov 03 23:51:03.649930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:03.633048    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:51:03.649930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:03.633324    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy" containerID="cri-o://f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb" gracePeriod=30
Nov 03 23:51:04.857480 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:04.857437    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:51:06.274138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:06.274099    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:51:08.880913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.880871    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:51:08.935672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.935622    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.935868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.935695    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.935936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.935894    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.935936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.935926    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936009    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936058    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936156 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936109    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936154    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936193    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\") pod \"27977e73-bc5d-4951-bf7a-35ef9f157302\" (UID: \"27977e73-bc5d-4951-bf7a-35ef9f157302\") "
Nov 03 23:51:08.936762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936699    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:08.936762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.936710    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:08.937221 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.937103    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:08.937221 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.937129    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:08.937508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.937477    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:08.938142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.938111    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:08.938236 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.938182    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:08.940433 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.940371    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "kube-api-access-57wjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:08.940542 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.940477    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:51:08.940613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.940575    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:08.942292 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.942261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token" pod "27977e73-bc5d-4951-bf7a-35ef9f157302" (UID: "27977e73-bc5d-4951-bf7a-35ef9f157302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:08.967227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:08.967202    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:51:09.038313 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038264    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038484    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038578    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038740    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038783    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.038962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038892    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.039154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.038969    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.039154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039066    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\") pod \"702cdb11-4642-4c8c-a286-adc7aa174473\" (UID: \"702cdb11-4642-4c8c-a286-adc7aa174473\") "
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039840    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wjx\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-kube-api-access-57wjx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039870    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039885    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039900    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039914    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27977e73-bc5d-4951-bf7a-35ef9f157302-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039927    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27977e73-bc5d-4951-bf7a-35ef9f157302-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.039941    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27977e73-bc5d-4951-bf7a-35ef9f157302-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.040405    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.040476    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:09.040552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.040523    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:09.041361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.041329    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:09.041547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.041520    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:09.044921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.044884    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:51:09.045517 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.045492    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "kube-api-access-xtkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:09.045779 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.045759    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:09.046023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.046006    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy" pod "702cdb11-4642-4c8c-a286-adc7aa174473" (UID: "702cdb11-4642-4c8c-a286-adc7aa174473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141805    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141840    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141851    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141860    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702cdb11-4642-4c8c-a286-adc7aa174473-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141869    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141878    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkzh\" (UniqueName: \"kubernetes.io/projected/702cdb11-4642-4c8c-a286-adc7aa174473-kube-api-access-xtkzh\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.141892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141898    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702cdb11-4642-4c8c-a286-adc7aa174473-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.142239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141907    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.142239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.141916    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702cdb11-4642-4c8c-a286-adc7aa174473-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:09.333463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.333422    6430 generic.go:356] "Generic (PLEG): container finished" podID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerID="5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1" exitCode=0
Nov 03 23:51:09.333878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.333858    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"
Nov 03 23:51:09.335367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.335339    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" event={"ID":"27977e73-bc5d-4951-bf7a-35ef9f157302","Type":"ContainerDied","Data":"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"}
Nov 03 23:51:09.335587 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.335570    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8" event={"ID":"27977e73-bc5d-4951-bf7a-35ef9f157302","Type":"ContainerDied","Data":"a5b4d6c7fd646ac910c131a435f43a4128b26fa5ebcd9d23e1a82fb022900045"}
Nov 03 23:51:09.335720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.335682    6430 scope.go:122] "RemoveContainer" containerID="5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"
Nov 03 23:51:09.338299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.338274    6430 generic.go:356] "Generic (PLEG): container finished" podID="702cdb11-4642-4c8c-a286-adc7aa174473" containerID="f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb" exitCode=0
Nov 03 23:51:09.341422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.338459    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" event={"ID":"702cdb11-4642-4c8c-a286-adc7aa174473","Type":"ContainerDied","Data":"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"}
Nov 03 23:51:09.346421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.338541    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"
Nov 03 23:51:09.353101 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.353072    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74" event={"ID":"702cdb11-4642-4c8c-a286-adc7aa174473","Type":"ContainerDied","Data":"86396d3c87e4720b0158e4efccf1acb84bc185ee508fcdae296feadc837a5c5a"}
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.433625    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.439375    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-f4n74"]
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.439698    6430 scope.go:122] "RemoveContainer" containerID="5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:09.440194    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1\": container with ID starting with 5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1 not found: ID does not exist" containerID="5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.440229    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1"} err="failed to get container status \"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1\": rpc error: code = NotFound desc = could not find container \"5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1\": container with ID starting with 5612c38f5d8e472f9332481ba55b360141c4a993c03503aa8d0d937c7f9774d1 not found: ID does not exist"
Nov 03 23:51:09.441833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.440251    6430 scope.go:122] "RemoveContainer" containerID="f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"
Nov 03 23:51:09.494596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.494564    6430 scope.go:122] "RemoveContainer" containerID="f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"
Nov 03 23:51:09.495029 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:09.495002    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb\": container with ID starting with f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb not found: ID does not exist" containerID="f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"
Nov 03 23:51:09.495193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.495042    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb"} err="failed to get container status \"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb\": rpc error: code = NotFound desc = could not find container \"f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb\": container with ID starting with f274ed1ca15bce5f12d1844e82b53695dbef1072033931a8dcf9c391ec1159fb not found: ID does not exist"
Nov 03 23:51:09.500142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.500074    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:51:09.506219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:09.506174    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-dj7g8"]
Nov 03 23:51:10.261301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.261266    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" path="/var/lib/kubelet/pods/27977e73-bc5d-4951-bf7a-35ef9f157302/volumes"
Nov 03 23:51:10.261952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.261933    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" path="/var/lib/kubelet/pods/702cdb11-4642-4c8c-a286-adc7aa174473/volumes"
Nov 03 23:51:10.363826 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.363792    6430 generic.go:356] "Generic (PLEG): container finished" podID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerID="21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c" exitCode=0
Nov 03 23:51:10.364009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.363860    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerDied","Data":"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"}
Nov 03 23:51:10.364335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.364321    6430 scope.go:122] "RemoveContainer" containerID="21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"
Nov 03 23:51:10.366018 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.365992    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:10.366082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.366067    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:10.366121 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:10.366106    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:11.383675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:11.383639    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerStarted","Data":"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"}
Nov 03 23:51:11.529415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:11.527795    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:51:11.529415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:11.527932    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:11.529415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:11.527983    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:12.174793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.174758    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:51:12.177775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.177339    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:51:12.592498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.589170    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:12.592498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.589254    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.926522    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.986840    6430 cpu_manager.go:414] "containerMap: removing container" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.986871    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.986898    6430 cpu_manager.go:414] "containerMap: removing container" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.986907    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.987100    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="27977e73-bc5d-4951-bf7a-35ef9f157302" containerName="istio-proxy"
Nov 03 23:51:12.989328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.987119    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="702cdb11-4642-4c8c-a286-adc7aa174473" containerName="istio-proxy"
Nov 03 23:51:12.999734 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:12.999700    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.035165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.017781    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5f5cd2-openshift-default-dockercfg-npj4b\""
Nov 03 23:51:13.036867 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.036828    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfrq\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.037102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.037081    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.037224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.037203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.037353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.037340    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.037673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.037651    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.037816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.037802    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.078414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.073477    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.078414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.073626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.078414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.073766    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.100631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.099451    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195229    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195316    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195350    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195475    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195611    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195753    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfrq\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195827    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.195876    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.197769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.196349    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.203111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.203074    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.224441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.221076    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.225790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.225757    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.226259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.226238    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.361106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.360988    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfrq\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.406422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.387712    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.406422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.388045    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.406422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.390831    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\") pod \"e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") " pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:13.544561 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.544516    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:13.545052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.544923    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:13.672047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:13.671955    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:14.275182 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.275138    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:14.275302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.275230    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:14.275302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.275271    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:14.278313 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.278287    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:51:14.462728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.462246    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerStarted","Data":"0e6f0ba718f5524fedfc696c277100f2d4cb124a3164d0a6cf4fe9430ae0dda8"}
Nov 03 23:51:14.462728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.462340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerStarted","Data":"caa286788f144cb6edc289358038ba8a9b6493688682cd0d65c915a8f2eec52d"}
Nov 03 23:51:14.491094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.491037    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podStartSLOduration=2.4910225 podStartE2EDuration="2.4910225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:14.488843374 +0000 UTC m=+1290.379905313" watchObservedRunningTime="2026-11-03 23:51:14.4910225 +0000 UTC m=+1290.382084434"
Nov 03 23:51:14.531475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.530789    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:14.531475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.530861    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:14.704412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.704236    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:14.787510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:14.783514    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:51:15.122413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.120594    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:51:15.122413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.120726    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.154297 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.154269    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-beffe3-serving-cert\""
Nov 03 23:51:15.187410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.187337    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.187647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.187530    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.188266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.187914    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fml\" (UniqueName: \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.188266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.188015    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.335597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.330581    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.335597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.334015    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.335597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.334095    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.337671 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.336941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fml\" (UniqueName: \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.339845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.338409    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.360354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.360304    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.415940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.415860    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fml\" (UniqueName: \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.416650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.416582    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.557413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.538170    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:15.557413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.548087    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:15.557413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.548232    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:15.658447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.641730    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:15.658447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.632919    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:15.691333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.691250    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:15.753412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:15.738878    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:16.086048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.085482    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:16.289875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.287854    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:16.529472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.528709    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:16.529472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.528778    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:16.567709 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.566632    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:51:16.570717 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:51:16.570689    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e83ac_ffe4_4d2c_ae12_bdd769f07b8f.slice/crio-c665760d43449e85a3e6495d06787710457208271bba178a761cc72fb50225e2 WatchSource:0}: Error finding container c665760d43449e85a3e6495d06787710457208271bba178a761cc72fb50225e2: Status 404 returned error can't find the container with id c665760d43449e85a3e6495d06787710457208271bba178a761cc72fb50225e2
Nov 03 23:51:16.573379 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.573325    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:16.574927 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:16.574700    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:51:17.182933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.182883    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" podUID="211ddd11-9302-4c85-9330-0269815f2af4" containerName="payload-pre-processing" containerID="cri-o://9de4ecd8c05f6a0d4d1e9c850bdab24d17680f7c8b68bdbef2789a7dd1244ed8" gracePeriod=25
Nov 03 23:51:17.183736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.183702    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" podUID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" containerName="payload-processing" containerID="cri-o://7777702e086fe9f2dbdd8694b798ade95ba36933dc139e5bda9297e9929f3b09" gracePeriod=25
Nov 03 23:51:17.535784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.535271    6430 patch_prober.go:29] interesting pod/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 23:51:17.535784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.535333    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 23:51:17.597024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.596974    6430 generic.go:356] "Generic (PLEG): container finished" podID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" containerID="7777702e086fe9f2dbdd8694b798ade95ba36933dc139e5bda9297e9929f3b09" exitCode=0
Nov 03 23:51:17.597365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.597340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" event={"ID":"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e","Type":"ContainerDied","Data":"7777702e086fe9f2dbdd8694b798ade95ba36933dc139e5bda9297e9929f3b09"}
Nov 03 23:51:17.607554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.600631    6430 generic.go:356] "Generic (PLEG): container finished" podID="211ddd11-9302-4c85-9330-0269815f2af4" containerID="9de4ecd8c05f6a0d4d1e9c850bdab24d17680f7c8b68bdbef2789a7dd1244ed8" exitCode=0
Nov 03 23:51:17.607554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.600783    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" event={"ID":"211ddd11-9302-4c85-9330-0269815f2af4","Type":"ContainerDied","Data":"9de4ecd8c05f6a0d4d1e9c850bdab24d17680f7c8b68bdbef2789a7dd1244ed8"}
Nov 03 23:51:17.607554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.603627    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" event={"ID":"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f","Type":"ContainerStarted","Data":"c665760d43449e85a3e6495d06787710457208271bba178a761cc72fb50225e2"}
Nov 03 23:51:17.860621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.860589    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:51:17.895408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.895335    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\" (UniqueName: \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\") pod \"211ddd11-9302-4c85-9330-0269815f2af4\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") "
Nov 03 23:51:17.895563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.895425    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\") pod \"211ddd11-9302-4c85-9330-0269815f2af4\" (UID: \"211ddd11-9302-4c85-9330-0269815f2af4\") "
Nov 03 23:51:17.896902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.896874    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume" pod "211ddd11-9302-4c85-9330-0269815f2af4" (UID: "211ddd11-9302-4c85-9330-0269815f2af4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:17.899940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.899033    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx" pod "211ddd11-9302-4c85-9330-0269815f2af4" (UID: "211ddd11-9302-4c85-9330-0269815f2af4"). InnerVolumeSpecName "kube-api-access-sftgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:17.950414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.947741    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:51:17.999451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.997358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\" (UniqueName: \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\") pod \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") "
Nov 03 23:51:17.999451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.997897    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\") pod \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\" (UID: \"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e\") "
Nov 03 23:51:17.999451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.998514    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftgx\" (UniqueName: \"kubernetes.io/projected/211ddd11-9302-4c85-9330-0269815f2af4-kube-api-access-sftgx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:17.999451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.998538    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211ddd11-9302-4c85-9330-0269815f2af4-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:17.999451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:17.998899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume" pod "d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" (UID: "d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:18.006320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.004802    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp" pod "d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" (UID: "d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e"). InnerVolumeSpecName "kube-api-access-22pfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:18.183428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.175138    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22pfp\" (UniqueName: \"kubernetes.io/projected/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-kube-api-access-22pfp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:18.183428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.175182    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:18.258963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.258932    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:18.269604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.269543    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" containerName="payload-processing"
Nov 03 23:51:18.273503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.273474    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" containerName="payload-processing"
Nov 03 23:51:18.273792 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.273775    6430 cpu_manager.go:414] "containerMap: removing container" podUID="211ddd11-9302-4c85-9330-0269815f2af4" containerName="payload-pre-processing"
Nov 03 23:51:18.279833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.279804    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211ddd11-9302-4c85-9330-0269815f2af4" containerName="payload-pre-processing"
Nov 03 23:51:18.280120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.280100    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="211ddd11-9302-4c85-9330-0269815f2af4" containerName="payload-pre-processing"
Nov 03 23:51:18.280211 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.280134    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" containerName="payload-processing"
Nov 03 23:51:18.296430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.296405    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:18.297696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.297670    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.304185 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.304162    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5f5cd2\""
Nov 03 23:51:18.319259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.319221    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:18.319375 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.319277    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:18.319597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.319577    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.324903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.324874    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5f5cd2-dockercfg-nxg8g\""
Nov 03 23:51:18.579682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.567931    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:51:18.581786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.581475    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:51:18.593627 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.592762    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:51:18.626674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.626636    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75nw\" (UniqueName: \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.627093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.627069    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.627294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.627279    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5l7\" (UniqueName: \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.638649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.638616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.679665 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.679624    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85" event={"ID":"d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e","Type":"ContainerDied","Data":"b3c60e2e1f4a837f9e7bf56f6362d5b67abf870169ef675842ea7bdd8548c146"}
Nov 03 23:51:18.679864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.679698    6430 scope.go:122] "RemoveContainer" containerID="7777702e086fe9f2dbdd8694b798ade95ba36933dc139e5bda9297e9929f3b09"
Nov 03 23:51:18.679946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.679871    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"
Nov 03 23:51:18.711598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.711562    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn" event={"ID":"211ddd11-9302-4c85-9330-0269815f2af4","Type":"ContainerDied","Data":"ef92c7e8be7f87880e5214a82da8669f111be120e9293f9b7cdeebf26f806370"}
Nov 03 23:51:18.722426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.721502    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"
Nov 03 23:51:18.730946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.729297    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:51:18.731566 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.731544    6430 scope.go:122] "RemoveContainer" containerID="9de4ecd8c05f6a0d4d1e9c850bdab24d17680f7c8b68bdbef2789a7dd1244ed8"
Nov 03 23:51:18.736826 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.736797    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" event={"ID":"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f","Type":"ContainerStarted","Data":"5609a2bbe71ff562c029bb70b8d76c6e41a414b70da7687ffccdfb18c32dd27a"}
Nov 03 23:51:18.737362 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.737341    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:18.742158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.742135    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-532e80-66fd4967dd-wmt85"]
Nov 03 23:51:18.756790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.756761    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.756913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.756862    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5l7\" (UniqueName: \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.756913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.756896    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.757170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.757150    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75nw\" (UniqueName: \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.757855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.757831    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.757945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.757868    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.773835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.773782    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5l7\" (UniqueName: \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\") pod \"payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.778327 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.778299    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75nw\" (UniqueName: \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\") pod \"payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:18.785293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.785241    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" podStartSLOduration=3.969375279 podStartE2EDuration="4.785222766s" totalImagesPullingTime="815.847487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:18.763703772 +0000 UTC m=+1294.654765706" watchObservedRunningTime="2026-11-03 23:51:18.785222766 +0000 UTC m=+1294.676284700"
Nov 03 23:51:18.785864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.785840    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:51:18.786126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.786104    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:18.791571 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.791549    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-532e80-5745fd46c-n9sbn"]
Nov 03 23:51:18.930321 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:18.928939    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:19.130444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.129095    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:19.137901 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:51:19.137848    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7c4e63_1a02_4619_b2f2_f23d88202e4c.slice/crio-574352fc093623fc8a646968e87ec94f2549e90a06726cd2db10d40e56827552 WatchSource:0}: Error finding container 574352fc093623fc8a646968e87ec94f2549e90a06726cd2db10d40e56827552: Status 404 returned error can't find the container with id 574352fc093623fc8a646968e87ec94f2549e90a06726cd2db10d40e56827552
Nov 03 23:51:19.374912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.374885    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:19.376444 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:51:19.376418    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4dc0bd_b2b6_4db5_a169_dcc5fb2a50ea.slice/crio-c17f65edfa38e235f8942d48f7f914a6bfe38cfa19544bacffa1b0076bee25f9 WatchSource:0}: Error finding container c17f65edfa38e235f8942d48f7f914a6bfe38cfa19544bacffa1b0076bee25f9: Status 404 returned error can't find the container with id c17f65edfa38e235f8942d48f7f914a6bfe38cfa19544bacffa1b0076bee25f9
Nov 03 23:51:19.527727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.524715    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:51:19.527727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.524969    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" podUID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" containerName="maas-api" containerID="cri-o://98191d2906ece4a22d1583c71fa674e1085038df0be5efcb32659dbeb8d16472" gracePeriod=30
Nov 03 23:51:19.895509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.895415    6430 generic.go:356] "Generic (PLEG): container finished" podID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" containerID="98191d2906ece4a22d1583c71fa674e1085038df0be5efcb32659dbeb8d16472" exitCode=0
Nov 03 23:51:19.895832 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.895806    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" event={"ID":"e95a1028-5b73-4b08-9b04-d57f8ad646a5","Type":"ContainerDied","Data":"98191d2906ece4a22d1583c71fa674e1085038df0be5efcb32659dbeb8d16472"}
Nov 03 23:51:19.905152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.904100    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" event={"ID":"5c7c4e63-1a02-4619-b2f2-f23d88202e4c","Type":"ContainerStarted","Data":"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"}
Nov 03 23:51:19.905152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.904168    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" event={"ID":"5c7c4e63-1a02-4619-b2f2-f23d88202e4c","Type":"ContainerStarted","Data":"574352fc093623fc8a646968e87ec94f2549e90a06726cd2db10d40e56827552"}
Nov 03 23:51:19.905400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.905183    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:19.911160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.910364    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" event={"ID":"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea","Type":"ContainerStarted","Data":"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"}
Nov 03 23:51:19.911160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.910442    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" event={"ID":"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea","Type":"ContainerStarted","Data":"c17f65edfa38e235f8942d48f7f914a6bfe38cfa19544bacffa1b0076bee25f9"}
Nov 03 23:51:19.911160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:19.910975    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:20.014482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.008579    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" podStartSLOduration=2.008559971 podStartE2EDuration="2.008559971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:20.003823805 +0000 UTC m=+1295.894885743" watchObservedRunningTime="2026-11-03 23:51:20.008559971 +0000 UTC m=+1295.899621905"
Nov 03 23:51:20.067476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.067447    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:51:20.101601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.101521    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" podStartSLOduration=2.101500861 podStartE2EDuration="2.101500861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:20.052579913 +0000 UTC m=+1295.943641848" watchObservedRunningTime="2026-11-03 23:51:20.101500861 +0000 UTC m=+1295.992562804"
Nov 03 23:51:20.258547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.256360    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\" (UniqueName: \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\") pod \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") "
Nov 03 23:51:20.258547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.256468    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\") pod \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") "
Nov 03 23:51:20.258547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.256646    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\") pod \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") "
Nov 03 23:51:20.258547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.256732    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\") pod \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\" (UID: \"e95a1028-5b73-4b08-9b04-d57f8ad646a5\") "
Nov 03 23:51:20.263483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.261875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca" pod "e95a1028-5b73-4b08-9b04-d57f8ad646a5" (UID: "e95a1028-5b73-4b08-9b04-d57f8ad646a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:20.282420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.280584    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs" pod "e95a1028-5b73-4b08-9b04-d57f8ad646a5" (UID: "e95a1028-5b73-4b08-9b04-d57f8ad646a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:51:20.283805 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.283775    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211ddd11-9302-4c85-9330-0269815f2af4" path="/var/lib/kubelet/pods/211ddd11-9302-4c85-9330-0269815f2af4/volumes"
Nov 03 23:51:20.284796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.284772    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e" path="/var/lib/kubelet/pods/d131d3e1-a905-4b7f-9ce9-c8f5e75deb8e/volumes"
Nov 03 23:51:20.305446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.301843    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls" pod "e95a1028-5b73-4b08-9b04-d57f8ad646a5" (UID: "e95a1028-5b73-4b08-9b04-d57f8ad646a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:51:20.305924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.305840    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml" pod "e95a1028-5b73-4b08-9b04-d57f8ad646a5" (UID: "e95a1028-5b73-4b08-9b04-d57f8ad646a5"). InnerVolumeSpecName "kube-api-access-dlzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:20.359194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.359146    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzml\" (UniqueName: \"kubernetes.io/projected/e95a1028-5b73-4b08-9b04-d57f8ad646a5-kube-api-access-dlzml\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:20.359194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.359177    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:20.359194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.359188    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e95a1028-5b73-4b08-9b04-d57f8ad646a5-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:20.359194 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.359200    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95a1028-5b73-4b08-9b04-d57f8ad646a5-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:20.430569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.430526    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:51:20.431040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.430605    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:20.953415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.943464    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f" event={"ID":"e95a1028-5b73-4b08-9b04-d57f8ad646a5","Type":"ContainerDied","Data":"595b0b9bc88be77395558df83984fbe2d0aa6e5bc89b4547c02b491c1f35f967"}
Nov 03 23:51:20.953415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.943547    6430 scope.go:122] "RemoveContainer" containerID="98191d2906ece4a22d1583c71fa674e1085038df0be5efcb32659dbeb8d16472"
Nov 03 23:51:20.953415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:20.943704    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"
Nov 03 23:51:21.010293 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.010259    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:51:21.015262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.015226    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-532e80-6c65c6b45c-c642f"]
Nov 03 23:51:21.295904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.295855    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" podUID="8e82018a-1491-47aa-b218-68a35f1de35c" containerName="main" containerID="cri-o://27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5" gracePeriod=45
Nov 03 23:51:21.621411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.616623    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:51:21.706064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706020    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.706260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706143    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.706260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706196    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\" (UniqueName: \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.706260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706224    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.706463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706326    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.706463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.706380    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\") pod \"8e82018a-1491-47aa-b218-68a35f1de35c\" (UID: \"8e82018a-1491-47aa-b218-68a35f1de35c\") "
Nov 03 23:51:21.709030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.708935    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:21.709030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.709004    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:21.709878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.709856    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:21.714749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.714713    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "kube-api-access-hxqwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:21.715021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.714993    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:51:21.716834 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.716799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm" pod "8e82018a-1491-47aa-b218-68a35f1de35c" (UID: "8e82018a-1491-47aa-b218-68a35f1de35c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808772    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808809    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808818    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqwl\" (UniqueName: \"kubernetes.io/projected/8e82018a-1491-47aa-b218-68a35f1de35c-kube-api-access-hxqwl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808827    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808835    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e82018a-1491-47aa-b218-68a35f1de35c-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.808858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.808844    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e82018a-1491-47aa-b218-68a35f1de35c-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:21.952557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.952523    6430 generic.go:356] "Generic (PLEG): container finished" podID="8e82018a-1491-47aa-b218-68a35f1de35c" containerID="27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5" exitCode=0
Nov 03 23:51:21.952746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.952576    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"
Nov 03 23:51:21.952746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.952610    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" event={"ID":"8e82018a-1491-47aa-b218-68a35f1de35c","Type":"ContainerDied","Data":"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"}
Nov 03 23:51:21.952746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.952666    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2" event={"ID":"8e82018a-1491-47aa-b218-68a35f1de35c","Type":"ContainerDied","Data":"dd9b086526f037731d9085c360b6e181b50660adfbec37b37a8ac4ba652161e3"}
Nov 03 23:51:21.952746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.952688    6430 scope.go:122] "RemoveContainer" containerID="27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"
Nov 03 23:51:21.963607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.963583    6430 scope.go:122] "RemoveContainer" containerID="27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"
Nov 03 23:51:21.963905 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:21.963884    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5\": container with ID starting with 27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5 not found: ID does not exist" containerID="27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"
Nov 03 23:51:21.963977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.963914    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5"} err="failed to get container status \"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5\": rpc error: code = NotFound desc = could not find container \"27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5\": container with ID starting with 27b69538701458584f7409a95e2476f7014914bd3b245e600225661d9e0a80c5 not found: ID does not exist"
Nov 03 23:51:21.980932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.980902    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:51:21.987492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:21.987464    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e4892d/ipp-route-e4892d-kserve-67b9bb5cd9-vfbh2"]
Nov 03 23:51:22.234991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:22.234897    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e82018a-1491-47aa-b218-68a35f1de35c" path="/var/lib/kubelet/pods/8e82018a-1491-47aa-b218-68a35f1de35c/volumes"
Nov 03 23:51:22.235427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:22.235368    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" path="/var/lib/kubelet/pods/e95a1028-5b73-4b08-9b04-d57f8ad646a5/volumes"
Nov 03 23:51:23.515086 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.515052    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:51:23.515638 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.515265    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy" containerID="cri-o://3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f" gracePeriod=30
Nov 03 23:51:23.973278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.973245    6430 generic.go:356] "Generic (PLEG): container finished" podID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerID="2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1" exitCode=0
Nov 03 23:51:23.973486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.973323    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerDied","Data":"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"}
Nov 03 23:51:23.973842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.973823    6430 scope.go:122] "RemoveContainer" containerID="2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"
Nov 03 23:51:23.975602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.975569    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:23.975721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.975657    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:23.975721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:23.975705    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:24.982649 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:24.982617    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerStarted","Data":"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"}
Nov 03 23:51:25.591345 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:25.591307    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:51:25.591631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:25.591609    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:25.591686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:25.591656    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:25.949707 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:25.949629    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:51:26.591695 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:26.591656    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:26.592210 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:26.591713    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:27.591209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:27.591165    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:27.591428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:27.591235    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:28.491758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.487198    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509130    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" containerName="maas-api"
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509161    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" containerName="maas-api"
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509244    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8e82018a-1491-47aa-b218-68a35f1de35c" containerName="main"
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509252    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e82018a-1491-47aa-b218-68a35f1de35c" containerName="main"
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509490    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e82018a-1491-47aa-b218-68a35f1de35c" containerName="main"
Nov 03 23:51:28.516410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.509516    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95a1028-5b73-4b08-9b04-d57f8ad646a5" containerName="maas-api"
Nov 03 23:51:28.541409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.538304    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:51:28.541409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.538525    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.552329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.550476    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5f5cd2-serving-cert\""
Nov 03 23:51:28.748910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.718176    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.748910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.718243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.748910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.718320    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4n8g\" (UniqueName: \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.748910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.718368    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.855424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.834522    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.855424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.834597    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.855424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.834698    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4n8g\" (UniqueName: \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.855424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.834757    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.912464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.900453    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.992645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.982042    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4n8g\" (UniqueName: \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:28.999126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:28.998773    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:29.002052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.001540    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:29.002052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.001641    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:29.002052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.001690    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:29.145855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.145828    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:51:29.313410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.310550    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.330762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.330719    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:29.399130 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.399041    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413607    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413683    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413786    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413847    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413889    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.413971    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.414003    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.414113    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\") pod \"f23b7462-87c4-403d-baa4-7f6bc47ac435\" (UID: \"f23b7462-87c4-403d-baa4-7f6bc47ac435\") "
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.414754    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23b7462-87c4-403d-baa4-7f6bc47ac435-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.415720    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.415899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:29.424414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.416068    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:29.425756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.425727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:29.437514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.430204    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:51:29.448506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.442701    6430 generic.go:356] "Generic (PLEG): container finished" podID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerID="3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f" exitCode=0
Nov 03 23:51:29.448506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.442912    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerDied","Data":"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"}
Nov 03 23:51:29.448506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.442973    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf" event={"ID":"f23b7462-87c4-403d-baa4-7f6bc47ac435","Type":"ContainerDied","Data":"cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb"}
Nov 03 23:51:29.448506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.443008    6430 scope.go:122] "RemoveContainer" containerID="3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"
Nov 03 23:51:29.448506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.443353    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"
Nov 03 23:51:29.451681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.451646    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:29.451910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.451892    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "kube-api-access-h8zq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:29.452700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.452674    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo" pod "f23b7462-87c4-403d-baa4-7f6bc47ac435" (UID: "f23b7462-87c4-403d-baa4-7f6bc47ac435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:51:29.460514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.460487    6430 scope.go:122] "RemoveContainer" containerID="29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"
Nov 03 23:51:29.470587 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.470566    6430 scope.go:122] "RemoveContainer" containerID="3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"
Nov 03 23:51:29.470893 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:29.470871    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f\": container with ID starting with 3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f not found: ID does not exist" containerID="3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"
Nov 03 23:51:29.470965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.470900    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f"} err="failed to get container status \"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f\": rpc error: code = NotFound desc = could not find container \"3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f\": container with ID starting with 3a92e7f237e5ddbb548f66e46f70a6fb3a4cbc270706ed10e981c4a49e81f20f not found: ID does not exist"
Nov 03 23:51:29.470965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.470920    6430 scope.go:122] "RemoveContainer" containerID="29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"
Nov 03 23:51:29.471203 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:29.471183    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e\": container with ID starting with 29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e not found: ID does not exist" containerID="29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"
Nov 03 23:51:29.471255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.471210    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e"} err="failed to get container status \"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e\": rpc error: code = NotFound desc = could not find container \"29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e\": container with ID starting with 29a0d2cfa611a45987fa1e9bdcd4e6a2d8ae086f2609d36c2a0875b08ad0ec5e not found: ID does not exist"
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518253    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518298    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518315    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518331    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518346    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518361    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23b7462-87c4-403d-baa4-7f6bc47ac435-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518376    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zq6\" (UniqueName: \"kubernetes.io/projected/f23b7462-87c4-403d-baa4-7f6bc47ac435-kube-api-access-h8zq6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.526410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.518412    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23b7462-87c4-403d-baa4-7f6bc47ac435-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:29.591356 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.591322    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:29.591531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.591377    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:29.829529 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.805854    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:51:29.934163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.934105    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:51:30.017427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.979898    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:51:30.017427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:29.994960    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:51:30.017427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:30.000609    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-532e80-openshift-default-549f6f5559-mp7qf"]
Nov 03 23:51:30.402136 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:30.401467    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23b7462_87c4_403d_baa4_7f6bc47ac435.slice/crio-cd87da188c5178b20b01544688d8389059cc5885f1d4cba06a0c741a17be72cb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23b7462_87c4_403d_baa4_7f6bc47ac435.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 23:51:30.406574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:30.406541    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" path="/var/lib/kubelet/pods/f23b7462-87c4-403d-baa4-7f6bc47ac435/volumes"
Nov 03 23:51:30.451162 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:30.451117    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" event={"ID":"edbb5285-0ece-4cc8-80f4-ca40ba39567f","Type":"ContainerStarted","Data":"6571dd8d6ffc7443fe8eb8be73693bdff64e6d95bdd3044ae28e9b61eac564af"}
Nov 03 23:51:30.591139 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:30.591097    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 23:51:30.591671 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:30.591156    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 23:51:31.460338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.460299    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" event={"ID":"edbb5285-0ece-4cc8-80f4-ca40ba39567f","Type":"ContainerStarted","Data":"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"}
Nov 03 23:51:31.460578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.460439    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:31.482300 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.482253    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" podStartSLOduration=2.716543078 podStartE2EDuration="3.482239264s" totalImagesPullingTime="765.696186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:31.48193643 +0000 UTC m=+1307.372998392" watchObservedRunningTime="2026-11-03 23:51:31.482239264 +0000 UTC m=+1307.373301201"
Nov 03 23:51:31.618161 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.613909    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:51:31.618161 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.615551    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:51:31.633410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:31.629189    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:51:32.038083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:32.030314    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:32.038083 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:32.031717    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:34.807512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:34.807469    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" podUID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" containerName="payload-processing" containerID="cri-o://30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0" gracePeriod=25
Nov 03 23:51:34.836942 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:34.836908    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:51:34.935922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:34.935857    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" podUID="c9361512-357a-47b6-97e9-199e67f673af" containerName="payload-pre-processing" containerID="cri-o://256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4" gracePeriod=25
Nov 03 23:51:35.096301 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.096267    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:51:35.187108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.186443    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\") pod \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") "
Nov 03 23:51:35.187108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.186542    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\" (UniqueName: \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\") pod \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\" (UID: \"cfcfa3a3-89bf-4fef-8969-69046c44a1d4\") "
Nov 03 23:51:35.187108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.186965    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume" pod "cfcfa3a3-89bf-4fef-8969-69046c44a1d4" (UID: "cfcfa3a3-89bf-4fef-8969-69046c44a1d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:35.187522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.187483    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.198447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.195589    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd" pod "cfcfa3a3-89bf-4fef-8969-69046c44a1d4" (UID: "cfcfa3a3-89bf-4fef-8969-69046c44a1d4"). InnerVolumeSpecName "kube-api-access-dxnmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:35.270873 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.270542    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:51:35.289056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.289016    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnmd\" (UniqueName: \"kubernetes.io/projected/cfcfa3a3-89bf-4fef-8969-69046c44a1d4-kube-api-access-dxnmd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.390664 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.390572    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\") pod \"c9361512-357a-47b6-97e9-199e67f673af\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") "
Nov 03 23:51:35.390664 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.390624    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\" (UniqueName: \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\") pod \"c9361512-357a-47b6-97e9-199e67f673af\" (UID: \"c9361512-357a-47b6-97e9-199e67f673af\") "
Nov 03 23:51:35.391019 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.390995    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume" pod "c9361512-357a-47b6-97e9-199e67f673af" (UID: "c9361512-357a-47b6-97e9-199e67f673af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:35.391164 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.391148    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9361512-357a-47b6-97e9-199e67f673af-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.392987 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.392961    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z" pod "c9361512-357a-47b6-97e9-199e67f673af" (UID: "c9361512-357a-47b6-97e9-199e67f673af"). InnerVolumeSpecName "kube-api-access-xg89z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:35.493142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.493102    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg89z\" (UniqueName: \"kubernetes.io/projected/c9361512-357a-47b6-97e9-199e67f673af-kube-api-access-xg89z\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:35.501487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.501455    6430 generic.go:356] "Generic (PLEG): container finished" podID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" containerID="30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0" exitCode=0
Nov 03 23:51:35.501672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.501511    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"
Nov 03 23:51:35.501672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.501536    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" event={"ID":"cfcfa3a3-89bf-4fef-8969-69046c44a1d4","Type":"ContainerDied","Data":"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"}
Nov 03 23:51:35.501672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.501595    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8" event={"ID":"cfcfa3a3-89bf-4fef-8969-69046c44a1d4","Type":"ContainerDied","Data":"77c97cbd40e629acb56d47ec478fd4397cb4dd97dfd5c59bea2c242e330ca1b2"}
Nov 03 23:51:35.501672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.501619    6430 scope.go:122] "RemoveContainer" containerID="30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"
Nov 03 23:51:35.503219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.503186    6430 generic.go:356] "Generic (PLEG): container finished" podID="c9361512-357a-47b6-97e9-199e67f673af" containerID="256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4" exitCode=0
Nov 03 23:51:35.503333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.503237    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"
Nov 03 23:51:35.503333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.503233    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" event={"ID":"c9361512-357a-47b6-97e9-199e67f673af","Type":"ContainerDied","Data":"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"}
Nov 03 23:51:35.503522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.503355    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g" event={"ID":"c9361512-357a-47b6-97e9-199e67f673af","Type":"ContainerDied","Data":"8f1eaf11b407b284f6fc542fca302e8bd56670bee4e18f3f47f8771af84bbc05"}
Nov 03 23:51:35.513069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.513013    6430 scope.go:122] "RemoveContainer" containerID="30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"
Nov 03 23:51:35.513425 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:35.513370    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0\": container with ID starting with 30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0 not found: ID does not exist" containerID="30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"
Nov 03 23:51:35.513531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.513437    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0"} err="failed to get container status \"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0\": rpc error: code = NotFound desc = could not find container \"30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0\": container with ID starting with 30e6b1ea11b6ff4c9ee0b3cfb90561b43b317b655c3741f75d576470b52ae6a0 not found: ID does not exist"
Nov 03 23:51:35.513531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.513464    6430 scope.go:122] "RemoveContainer" containerID="256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"
Nov 03 23:51:35.523554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.523530    6430 scope.go:122] "RemoveContainer" containerID="256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"
Nov 03 23:51:35.523837 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:35.523812    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4\": container with ID starting with 256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4 not found: ID does not exist" containerID="256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"
Nov 03 23:51:35.523885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.523848    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4"} err="failed to get container status \"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4\": rpc error: code = NotFound desc = could not find container \"256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4\": container with ID starting with 256f663fcc2fb5728939f36d878e23c340763e3222dffba8295003cb0872f7f4 not found: ID does not exist"
Nov 03 23:51:35.526711 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.526692    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:51:35.533045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.533010    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:51:35.538985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.538966    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c83d52-868cfd9b67-cq88g"]
Nov 03 23:51:35.559995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.559971    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:51:35.571855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:35.571824    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c83d52-575cbcd794-9mgl8"]
Nov 03 23:51:36.233427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:36.233337    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9361512-357a-47b6-97e9-199e67f673af" path="/var/lib/kubelet/pods/c9361512-357a-47b6-97e9-199e67f673af/volumes"
Nov 03 23:51:36.233820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:36.233792    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" path="/var/lib/kubelet/pods/cfcfa3a3-89bf-4fef-8969-69046c44a1d4/volumes"
Nov 03 23:51:37.473999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:37.473965    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:51:41.969844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.969801    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:51:41.982681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.982647    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:41.982906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.982890    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:41.983016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.983004    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" containerName="payload-processing"
Nov 03 23:51:41.983102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.983090    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" containerName="payload-processing"
Nov 03 23:51:42.028124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:41.983249    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c9361512-357a-47b6-97e9-199e67f673af" containerName="payload-pre-processing"
Nov 03 23:51:42.028340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.028316    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9361512-357a-47b6-97e9-199e67f673af" containerName="payload-pre-processing"
Nov 03 23:51:42.028731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.028713    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:42.028857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.028846    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:42.028947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.028936    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcfa3a3-89bf-4fef-8969-69046c44a1d4" containerName="payload-processing"
Nov 03 23:51:42.029049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.029039    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9361512-357a-47b6-97e9-199e67f673af" containerName="payload-pre-processing"
Nov 03 23:51:42.259473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.243448    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:51:42.266632 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.266594    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.278855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.278830    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c7e425-openshift-default-dockercfg-h6fns\""
Nov 03 23:51:42.356043 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356003    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356060    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25sq\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356200    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356252    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356323    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356381 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356374    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356494    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356547    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.356596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.356588    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458476    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458533    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458556    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458645    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458716    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458853    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25sq\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.458941 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458870    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.459157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.459157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.458955    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.459157 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.459127    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.459323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.459284    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.459569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.459547    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.461172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.461149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.461255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.461225    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.490811 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.490457    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.491380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.491358    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25sq\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.586818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.586786    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:42.924240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.924206    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:51:42.926183 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:51:42.926144    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a1413c_dbae_41d8_b921_73036dfa63db.slice/crio-b4977defa21f9ee222a06d3f6480e19edddee8ff26453f286bd9b1360b4815a4 WatchSource:0}: Error finding container b4977defa21f9ee222a06d3f6480e19edddee8ff26453f286bd9b1360b4815a4: Status 404 returned error can't find the container with id b4977defa21f9ee222a06d3f6480e19edddee8ff26453f286bd9b1360b4815a4
Nov 03 23:51:42.929593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.929563    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:42.929690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.929632    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:42.929690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:42.929672    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:43.651425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:43.651368    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" event={"ID":"62a1413c-dbae-41d8-b921-73036dfa63db","Type":"ContainerStarted","Data":"42f0ccb14f01eab83324f231ade72daf2ea35787c87f8e72bbecac58840b0d73"}
Nov 03 23:51:43.651902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:43.651441    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" event={"ID":"62a1413c-dbae-41d8-b921-73036dfa63db","Type":"ContainerStarted","Data":"b4977defa21f9ee222a06d3f6480e19edddee8ff26453f286bd9b1360b4815a4"}
Nov 03 23:51:43.701968 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:43.701900    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podStartSLOduration=2.7018802539999998 podStartE2EDuration="2.701880254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:43.699358516 +0000 UTC m=+1319.590420476" watchObservedRunningTime="2026-11-03 23:51:43.701880254 +0000 UTC m=+1319.592942189"
Nov 03 23:51:44.587257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:44.587219    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:44.588653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:44.588615    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:44.588784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:44.588678    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:45.001508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:45.001405    6430 scope.go:122] "RemoveContainer" containerID="4618b0e230d9b72721cd8b3a5e33183b5823e999cd88f01e17dcb0d05d127852"
Nov 03 23:51:45.588258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:45.588215    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:45.588477 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:45.588291    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:46.238102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:46.237612    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:51:46.238102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:46.237682    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:51:46.588117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:46.588068    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:46.588295 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:46.588135    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:47.176087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:47.176049    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:47.199208 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:47.198999    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:47.589412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:47.587849    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:47.589412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:47.587913    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:48.604412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:48.597033    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:48.604412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:48.597114    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:49.588098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.588048    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 03 23:51:49.588350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.588130    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 03 23:51:49.928572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.928534    6430 generic.go:356] "Generic (PLEG): container finished" podID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerID="d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09" exitCode=0
Nov 03 23:51:49.929082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.928610    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerDied","Data":"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"}
Nov 03 23:51:49.929129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.929084    6430 scope.go:122] "RemoveContainer" containerID="d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"
Nov 03 23:51:49.930799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.930770    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:49.930887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.930840    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:49.930887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:49.930871    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:51:50.592614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:50.592582    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:50.593124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:50.593101    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:50.593996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:50.593969    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:51:50.939992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:50.939900    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerStarted","Data":"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"}
Nov 03 23:51:51.033569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:51.033533    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:51.033811 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:51.033783    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:51.033900 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:51.033842    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:52.033895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.033860    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:52.034324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.033914    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:52.177366 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.177305    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" podUID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" containerName="payload-processing" containerID="cri-o://98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121" gracePeriod=25
Nov 03 23:51:52.199934 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.199885    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" podUID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" containerName="payload-pre-processing" containerID="cri-o://d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77" gracePeriod=25
Nov 03 23:51:52.689752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.689719    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:52.762302 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.762272    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\") pod \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") "
Nov 03 23:51:52.762371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.762327    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\" (UniqueName: \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\") pod \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\" (UID: \"5c7c4e63-1a02-4619-b2f2-f23d88202e4c\") "
Nov 03 23:51:52.764692 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.764377    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume" pod "5c7c4e63-1a02-4619-b2f2-f23d88202e4c" (UID: "5c7c4e63-1a02-4619-b2f2-f23d88202e4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:52.766056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.766023    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7" pod "5c7c4e63-1a02-4619-b2f2-f23d88202e4c" (UID: "5c7c4e63-1a02-4619-b2f2-f23d88202e4c"). InnerVolumeSpecName "kube-api-access-xp5l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:52.776520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.776498    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:52.864577 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.864541    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\") pod \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") "
Nov 03 23:51:52.864768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.864669    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\" (UniqueName: \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\") pod \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\" (UID: \"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea\") "
Nov 03 23:51:52.864924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.864899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume" pod "5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" (UID: "5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:51:52.865218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.865204    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:52.865268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.865222    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:52.865268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.865232    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5l7\" (UniqueName: \"kubernetes.io/projected/5c7c4e63-1a02-4619-b2f2-f23d88202e4c-kube-api-access-xp5l7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:52.866728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.866709    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw" pod "5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" (UID: "5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea"). InnerVolumeSpecName "kube-api-access-x75nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:51:52.955723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.955639    6430 generic.go:356] "Generic (PLEG): container finished" podID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" containerID="98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121" exitCode=0
Nov 03 23:51:52.955723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.955688    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"
Nov 03 23:51:52.955932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.955716    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" event={"ID":"5c7c4e63-1a02-4619-b2f2-f23d88202e4c","Type":"ContainerDied","Data":"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"}
Nov 03 23:51:52.955932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.955772    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m" event={"ID":"5c7c4e63-1a02-4619-b2f2-f23d88202e4c","Type":"ContainerDied","Data":"574352fc093623fc8a646968e87ec94f2549e90a06726cd2db10d40e56827552"}
Nov 03 23:51:52.955932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.955801    6430 scope.go:122] "RemoveContainer" containerID="98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"
Nov 03 23:51:52.957268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.957248    6430 generic.go:356] "Generic (PLEG): container finished" podID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" containerID="d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77" exitCode=0
Nov 03 23:51:52.957417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.957290    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"
Nov 03 23:51:52.957417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.957332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" event={"ID":"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea","Type":"ContainerDied","Data":"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"}
Nov 03 23:51:52.957417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.957403    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8" event={"ID":"5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea","Type":"ContainerDied","Data":"c17f65edfa38e235f8942d48f7f914a6bfe38cfa19544bacffa1b0076bee25f9"}
Nov 03 23:51:52.966229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.966197    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75nw\" (UniqueName: \"kubernetes.io/projected/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea-kube-api-access-x75nw\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:51:52.966684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.966666    6430 scope.go:122] "RemoveContainer" containerID="98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"
Nov 03 23:51:52.967001 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:52.966975    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121\": container with ID starting with 98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121 not found: ID does not exist" containerID="98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"
Nov 03 23:51:52.967107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.967006    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121"} err="failed to get container status \"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121\": rpc error: code = NotFound desc = could not find container \"98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121\": container with ID starting with 98c1969dde3a4df6c1340f6b4d3a2348bcdfc57f3f9513d2321872e010df9121 not found: ID does not exist"
Nov 03 23:51:52.967107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.967023    6430 scope.go:122] "RemoveContainer" containerID="d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"
Nov 03 23:51:52.977654 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.977631    6430 scope.go:122] "RemoveContainer" containerID="d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"
Nov 03 23:51:52.977920 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:51:52.977898    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77\": container with ID starting with d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77 not found: ID does not exist" containerID="d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"
Nov 03 23:51:52.977988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.977930    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77"} err="failed to get container status \"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77\": rpc error: code = NotFound desc = could not find container \"d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77\": container with ID starting with d884348565722025b9d0800e4525bab03f84466755e6e2d00b2706ba1a5bca77 not found: ID does not exist"
Nov 03 23:51:52.984704 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.984683    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:52.990485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:52.990463    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f5cd2-568584cc5f-4kjw8"]
Nov 03 23:51:53.003146 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.003123    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:53.032935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.032894    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-7df759bf88-rn74m"]
Nov 03 23:51:53.034621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.034597    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:53.034938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.034647    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:53.737412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.736507    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:51:53.737718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737689    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" containerName="payload-pre-processing"
Nov 03 23:51:53.737852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737719    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" containerName="payload-pre-processing"
Nov 03 23:51:53.737852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737745    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:53.737852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737753    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b7462-87c4-403d-baa4-7f6bc47ac435" containerName="istio-proxy"
Nov 03 23:51:53.737852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737770    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" containerName="payload-processing"
Nov 03 23:51:53.737852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737777    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" containerName="payload-processing"
Nov 03 23:51:53.738129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737956    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" containerName="payload-pre-processing"
Nov 03 23:51:53.738129 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.737976    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" containerName="payload-processing"
Nov 03 23:51:53.743954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.743907    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:51:53.744142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.744033    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:53.757432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.750630    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5f5cd2-dockercfg-ltjvz\""
Nov 03 23:51:53.757432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.750708    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5f5cd2\""
Nov 03 23:51:53.781912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.781874    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:53.782105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.781946    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftkb\" (UniqueName: \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:53.945429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.939480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:53.945429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.939612    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftkb\" (UniqueName: \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:53.945798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:53.945721    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:54.007528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.000780    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftkb\" (UniqueName: \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\") pod \"payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:54.049415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.048524    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:54.049415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.048608    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:54.074419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.066455    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:54.252749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.252707    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7c4e63-1a02-4619-b2f2-f23d88202e4c" path="/var/lib/kubelet/pods/5c7c4e63-1a02-4619-b2f2-f23d88202e4c/volumes"
Nov 03 23:51:54.253440 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.253370    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea" path="/var/lib/kubelet/pods/5e4dc0bd-b2b6-4db5-a169-dcc5fb2a50ea/volumes"
Nov 03 23:51:54.483193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:54.483159    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:51:54.485159 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:51:54.485134    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7753e801_ec56_435c_adf9_2e40490ee4e4.slice/crio-ec1bd26c73f29bee35490aff26f8e4658cea69cf593c154e1f8d8e11ac3d5517 WatchSource:0}: Error finding container ec1bd26c73f29bee35490aff26f8e4658cea69cf593c154e1f8d8e11ac3d5517: Status 404 returned error can't find the container with id ec1bd26c73f29bee35490aff26f8e4658cea69cf593c154e1f8d8e11ac3d5517
Nov 03 23:51:55.041520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.038793    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:55.041520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.038855    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:55.088506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.084377    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" event={"ID":"7753e801-ec56-435c-adf9-2e40490ee4e4","Type":"ContainerStarted","Data":"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"}
Nov 03 23:51:55.088506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.084456    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" event={"ID":"7753e801-ec56-435c-adf9-2e40490ee4e4","Type":"ContainerStarted","Data":"ec1bd26c73f29bee35490aff26f8e4658cea69cf593c154e1f8d8e11ac3d5517"}
Nov 03 23:51:55.088506 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.084628    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:51:55.105654 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:55.105572    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" podStartSLOduration=2.105554018 podStartE2EDuration="2.105554018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:51:55.104978189 +0000 UTC m=+1330.996040124" watchObservedRunningTime="2026-11-03 23:51:55.105554018 +0000 UTC m=+1330.996615954"
Nov 03 23:51:56.034084 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:56.034044    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 23:51:56.034291 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:56.034116    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 23:51:56.830518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:56.830481    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:51:56.831022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:56.830737    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" containerID="cri-o://04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856" gracePeriod=30
Nov 03 23:51:57.136470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:57.135826    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:57.140413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:57.137963    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:51:57.149408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:51:57.146690    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:52:01.617155 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:01.617114    6430 patch_prober.go:29] interesting pod/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:52:01.617568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:01.617186    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:52:02.119005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.118977    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:52:02.171440 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171381    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171528    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171723    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171774    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171812    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171868    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.171959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171915    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.172071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.171959    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.172071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172021    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\") pod \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\" (UID: \"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f\") "
Nov 03 23:52:02.172071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172003    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:02.172250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172227    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:02.172334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172253    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:02.172334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172316    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:02.172630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172606    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:02.172819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172799    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.172903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172826    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.172903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172844    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.172903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172860    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.172903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.172873    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.174324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.174296    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:02.174324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.174310    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:02.175254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.175230    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "kube-api-access-gg2qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:02.175919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.175894    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo" pod "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" (UID: "dc445bf6-97e7-46d7-9ae4-16a064f6aa4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:52:02.194277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.194221    6430 generic.go:356] "Generic (PLEG): container finished" podID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerID="04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856" exitCode=0
Nov 03 23:52:02.194526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.194487    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerDied","Data":"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"}
Nov 03 23:52:02.194624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.194564    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb" event={"ID":"dc445bf6-97e7-46d7-9ae4-16a064f6aa4f","Type":"ContainerDied","Data":"73401ef3ea16ca08b3e9756d9deb94b23c0309d7192d492dc866c7945ca3ac13"}
Nov 03 23:52:02.194624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.194611    6430 scope.go:122] "RemoveContainer" containerID="04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"
Nov 03 23:52:02.194849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.194817    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"
Nov 03 23:52:02.220028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.220003    6430 scope.go:122] "RemoveContainer" containerID="2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"
Nov 03 23:52:02.231051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.231027    6430 scope.go:122] "RemoveContainer" containerID="04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"
Nov 03 23:52:02.231489 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:02.231459    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856\": container with ID starting with 04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856 not found: ID does not exist" containerID="04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"
Nov 03 23:52:02.231580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.231498    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856"} err="failed to get container status \"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856\": rpc error: code = NotFound desc = could not find container \"04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856\": container with ID starting with 04051b2a1307a120396faa1c9eb9f1fe6f6375d961930d7295c32ec927b82856 not found: ID does not exist"
Nov 03 23:52:02.231580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.231516    6430 scope.go:122] "RemoveContainer" containerID="2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"
Nov 03 23:52:02.231870 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:02.231818    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1\": container with ID starting with 2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1 not found: ID does not exist" containerID="2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"
Nov 03 23:52:02.231915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.231881    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1"} err="failed to get container status \"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1\": rpc error: code = NotFound desc = could not find container \"2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1\": container with ID starting with 2e45f7bcdb63a9611c0cb7b5ba0a29c0fb75657882ed93797be372440e760be1 not found: ID does not exist"
Nov 03 23:52:02.234370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.234349    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:52:02.238112 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.238092    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c83d52-openshift-default-6465b9cc65-nz7jb"]
Nov 03 23:52:02.275031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.274989    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2qn\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-kube-api-access-gg2qn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.275031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.275017    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.275031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.275034    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:02.275260 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:02.275043    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:03.068892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:03.068856    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:52:03.083774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:03.083737    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:52:04.233014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:04.232979    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" path="/var/lib/kubelet/pods/dc445bf6-97e7-46d7-9ae4-16a064f6aa4f/volumes"
Nov 03 23:52:04.914234 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:04.914195    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:52:06.103140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:06.103110    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:52:07.684070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:07.684033    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:52:07.684534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:07.684328    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy" containerID="cri-o://57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb" gracePeriod=30
Nov 03 23:52:08.084819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.084505    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" podUID="e129e2dc-6051-45ee-9c70-ef61e9283038" containerName="payload-pre-processing" containerID="cri-o://fbd3a782c31f194e8f433d2622e9cc2f1090b2f108eeec02cc6f30b458e2f497" gracePeriod=25
Nov 03 23:52:08.204090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.204040    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" podUID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" containerName="payload-processing" containerID="cri-o://7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0" gracePeriod=25
Nov 03 23:52:08.444552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.443894    6430 generic.go:356] "Generic (PLEG): container finished" podID="e129e2dc-6051-45ee-9c70-ef61e9283038" containerID="fbd3a782c31f194e8f433d2622e9cc2f1090b2f108eeec02cc6f30b458e2f497" exitCode=0
Nov 03 23:52:08.446768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.446020    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" event={"ID":"e129e2dc-6051-45ee-9c70-ef61e9283038","Type":"ContainerDied","Data":"fbd3a782c31f194e8f433d2622e9cc2f1090b2f108eeec02cc6f30b458e2f497"}
Nov 03 23:52:08.585200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.585169    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:52:08.650135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.648612    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\") pod \"e129e2dc-6051-45ee-9c70-ef61e9283038\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") "
Nov 03 23:52:08.650135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.648664    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\" (UniqueName: \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\") pod \"e129e2dc-6051-45ee-9c70-ef61e9283038\" (UID: \"e129e2dc-6051-45ee-9c70-ef61e9283038\") "
Nov 03 23:52:08.650516 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.650321    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume" pod "e129e2dc-6051-45ee-9c70-ef61e9283038" (UID: "e129e2dc-6051-45ee-9c70-ef61e9283038"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:08.652370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.652327    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr" pod "e129e2dc-6051-45ee-9c70-ef61e9283038" (UID: "e129e2dc-6051-45ee-9c70-ef61e9283038"). InnerVolumeSpecName "kube-api-access-jr8hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:08.674858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.674827    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:52:08.750126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.750089    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\" (UniqueName: \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\") pod \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") "
Nov 03 23:52:08.750710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.750309    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\") pod \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\" (UID: \"eda6a7cc-7654-4aa7-b731-d6a1977340a8\") "
Nov 03 23:52:08.750782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.750723    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume" pod "eda6a7cc-7654-4aa7-b731-d6a1977340a8" (UID: "eda6a7cc-7654-4aa7-b731-d6a1977340a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:08.751073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.751051    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e129e2dc-6051-45ee-9c70-ef61e9283038-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:08.751073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.751073    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8hr\" (UniqueName: \"kubernetes.io/projected/e129e2dc-6051-45ee-9c70-ef61e9283038-kube-api-access-jr8hr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:08.751243 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.751087    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda6a7cc-7654-4aa7-b731-d6a1977340a8-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:08.753576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.753543    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7" pod "eda6a7cc-7654-4aa7-b731-d6a1977340a8" (UID: "eda6a7cc-7654-4aa7-b731-d6a1977340a8"). InnerVolumeSpecName "kube-api-access-dzww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:08.853094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:08.853059    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzww7\" (UniqueName: \"kubernetes.io/projected/eda6a7cc-7654-4aa7-b731-d6a1977340a8-kube-api-access-dzww7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:09.451975 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.451869    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"
Nov 03 23:52:09.451975 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.451887    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k" event={"ID":"e129e2dc-6051-45ee-9c70-ef61e9283038","Type":"ContainerDied","Data":"901bdfbb44bb9a6368af616e8b24708aa8270cecc9e8e7c5f393eb78c020c4ea"}
Nov 03 23:52:09.451975 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.451966    6430 scope.go:122] "RemoveContainer" containerID="fbd3a782c31f194e8f433d2622e9cc2f1090b2f108eeec02cc6f30b458e2f497"
Nov 03 23:52:09.453595 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.453573    6430 generic.go:356] "Generic (PLEG): container finished" podID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" containerID="7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0" exitCode=0
Nov 03 23:52:09.453731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.453614    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"
Nov 03 23:52:09.453731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.453653    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" event={"ID":"eda6a7cc-7654-4aa7-b731-d6a1977340a8","Type":"ContainerDied","Data":"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"}
Nov 03 23:52:09.453731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.453704    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm" event={"ID":"eda6a7cc-7654-4aa7-b731-d6a1977340a8","Type":"ContainerDied","Data":"f5d3df9a6306e2fe7ed28f2fd29f66a90a6139cb02a4958d0553983c66c954ae"}
Nov 03 23:52:09.464517 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.464494    6430 scope.go:122] "RemoveContainer" containerID="7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"
Nov 03 23:52:09.474346 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.474326    6430 scope.go:122] "RemoveContainer" containerID="7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"
Nov 03 23:52:09.474704 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:09.474685    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0\": container with ID starting with 7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0 not found: ID does not exist" containerID="7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"
Nov 03 23:52:09.474754 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.474715    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0"} err="failed to get container status \"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0\": rpc error: code = NotFound desc = could not find container \"7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0\": container with ID starting with 7038c5f14de851f900663ca6ebd17246d1897382b0f2613b4e02e16814092eb0 not found: ID does not exist"
Nov 03 23:52:09.484046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.484018    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:52:09.487044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.487011    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-695f03-55464db75f-xlg7k"]
Nov 03 23:52:09.503424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.499649    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:52:09.544994 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.544943    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-695f03-874474f46-pdwmm"]
Nov 03 23:52:09.548587 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.548554    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:52:09.548778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:09.548756    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" podUID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" containerName="maas-api" containerID="cri-o://dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e" gracePeriod=30
Nov 03 23:52:10.273141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.272770    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:52:10.274845 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.274817    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e129e2dc-6051-45ee-9c70-ef61e9283038" path="/var/lib/kubelet/pods/e129e2dc-6051-45ee-9c70-ef61e9283038/volumes"
Nov 03 23:52:10.275421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.275404    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" path="/var/lib/kubelet/pods/eda6a7cc-7654-4aa7-b731-d6a1977340a8/volumes"
Nov 03 23:52:10.355380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.355344    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\") pod \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") "
Nov 03 23:52:10.355602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.355420    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\" (UniqueName: \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\") pod \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") "
Nov 03 23:52:10.355602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.355480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\") pod \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") "
Nov 03 23:52:10.355602 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.355584    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\") pod \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\" (UID: \"f025ba1a-bdfa-4812-8e0c-1c7895147dc5\") "
Nov 03 23:52:10.356158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.355903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca" pod "f025ba1a-bdfa-4812-8e0c-1c7895147dc5" (UID: "f025ba1a-bdfa-4812-8e0c-1c7895147dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:10.356622 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.356551    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:10.358250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.358194    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls" pod "f025ba1a-bdfa-4812-8e0c-1c7895147dc5" (UID: "f025ba1a-bdfa-4812-8e0c-1c7895147dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:10.358353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.358301    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6" pod "f025ba1a-bdfa-4812-8e0c-1c7895147dc5" (UID: "f025ba1a-bdfa-4812-8e0c-1c7895147dc5"). InnerVolumeSpecName "kube-api-access-qmbv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:10.358426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.358344    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs" pod "f025ba1a-bdfa-4812-8e0c-1c7895147dc5" (UID: "f025ba1a-bdfa-4812-8e0c-1c7895147dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:10.563532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.544738    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:10.563532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.544789    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:10.563532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.544811    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbv6\" (UniqueName: \"kubernetes.io/projected/f025ba1a-bdfa-4812-8e0c-1c7895147dc5-kube-api-access-qmbv6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:10.643412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.631968    6430 generic.go:356] "Generic (PLEG): container finished" podID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" containerID="dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e" exitCode=0
Nov 03 23:52:10.646135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.646104    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"
Nov 03 23:52:10.688802 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.658725    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" event={"ID":"f025ba1a-bdfa-4812-8e0c-1c7895147dc5","Type":"ContainerDied","Data":"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"}
Nov 03 23:52:10.691101 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.690158    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd" event={"ID":"f025ba1a-bdfa-4812-8e0c-1c7895147dc5","Type":"ContainerDied","Data":"286b41d4ef3a8c46e5ac64b103763a280525ff427699ebe620d663b348f6e317"}
Nov 03 23:52:10.691101 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.690236    6430 scope.go:122] "RemoveContainer" containerID="dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"
Nov 03 23:52:10.728911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.728878    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:52:10.735806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.735778    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-695f03-7b89d8ffc8-745sd"]
Nov 03 23:52:10.736333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.736300    6430 scope.go:122] "RemoveContainer" containerID="dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"
Nov 03 23:52:10.737030 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:10.736999    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e\": container with ID starting with dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e not found: ID does not exist" containerID="dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"
Nov 03 23:52:10.737180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:10.737151    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e"} err="failed to get container status \"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e\": rpc error: code = NotFound desc = could not find container \"dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e\": container with ID starting with dc9e3886ab6d924c9644dcb0c5d85001f9503eac360a1f5658a4d5e26b08536e not found: ID does not exist"
Nov 03 23:52:12.242836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:12.242803    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" path="/var/lib/kubelet/pods/f025ba1a-bdfa-4812-8e0c-1c7895147dc5/volumes"
Nov 03 23:52:13.160748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.160713    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:52:13.235328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235294    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235336    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235359    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235465    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235525    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235567    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235685    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235741    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.235798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:13.236030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235803    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\") pod \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\" (UID: \"2c125e32-7920-4c23-aa7d-f9bc57baa92a\") "
Nov 03 23:52:13.236030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.235850    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:13.236142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236069    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:13.236341 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236318    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:13.236631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236606    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.236720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236636    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.236720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236652    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.236720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236665    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.236720 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.236670    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:13.238227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.238198    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "kube-api-access-qxpq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:13.238650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.238623    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:52:13.238770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.238746    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:13.238937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.238916    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy" pod "2c125e32-7920-4c23-aa7d-f9bc57baa92a" (UID: "2c125e32-7920-4c23-aa7d-f9bc57baa92a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:13.338285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.338200    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.338285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.338233    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.338285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.338243    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.338285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.338253    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpq7\" (UniqueName: \"kubernetes.io/projected/2c125e32-7920-4c23-aa7d-f9bc57baa92a-kube-api-access-qxpq7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.338285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.338265    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c125e32-7920-4c23-aa7d-f9bc57baa92a-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:13.710336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.710248    6430 generic.go:356] "Generic (PLEG): container finished" podID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerID="57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb" exitCode=0
Nov 03 23:52:13.710336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.710305    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"
Nov 03 23:52:13.710336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.710318    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerDied","Data":"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"}
Nov 03 23:52:13.710650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.710375    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn" event={"ID":"2c125e32-7920-4c23-aa7d-f9bc57baa92a","Type":"ContainerDied","Data":"a0251fad7fc18965a0c4bb842e45f8670793539aaeb8636f3d5644940c51bd64"}
Nov 03 23:52:13.710650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.710422    6430 scope.go:122] "RemoveContainer" containerID="57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"
Nov 03 23:52:13.726508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.726484    6430 scope.go:122] "RemoveContainer" containerID="132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"
Nov 03 23:52:13.738708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.738674    6430 scope.go:122] "RemoveContainer" containerID="57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"
Nov 03 23:52:13.739173 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:13.739040    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb\": container with ID starting with 57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb not found: ID does not exist" containerID="57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"
Nov 03 23:52:13.739173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.739081    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb"} err="failed to get container status \"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb\": rpc error: code = NotFound desc = could not find container \"57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb\": container with ID starting with 57301d21320d342e5e2ec2154d0be6582eb36bf22ba33bb6affdeb84b0ea9cfb not found: ID does not exist"
Nov 03 23:52:13.739173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.739105    6430 scope.go:122] "RemoveContainer" containerID="132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"
Nov 03 23:52:13.739912 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:13.739877    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8\": container with ID starting with 132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8 not found: ID does not exist" containerID="132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"
Nov 03 23:52:13.740020 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.739914    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8"} err="failed to get container status \"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8\": rpc error: code = NotFound desc = could not find container \"132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8\": container with ID starting with 132fb163edfa725e17c0ff94a0a3da2b520d1150fe99e7309c678dd38d6caaa8 not found: ID does not exist"
Nov 03 23:52:13.745454 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.745429    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:52:13.749806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:13.749781    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4892d-openshift-default-f99bf986c-b7psn"]
Nov 03 23:52:14.233495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:14.233453    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" path="/var/lib/kubelet/pods/2c125e32-7920-4c23-aa7d-f9bc57baa92a/volumes"
Nov 03 23:52:16.975253 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:16.973224    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": read tcp 10.134.0.2:48234->10.134.0.120:15021: read: connection reset by peer" start-of-body=
Nov 03 23:52:16.975253 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:16.973729    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": read tcp 10.134.0.2:48234->10.134.0.120:15021: read: connection reset by peer"
Nov 03 23:52:17.740146 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.740113    6430 generic.go:356] "Generic (PLEG): container finished" podID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerID="0e6f0ba718f5524fedfc696c277100f2d4cb124a3164d0a6cf4fe9430ae0dda8" exitCode=0
Nov 03 23:52:17.740360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.740158    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerDied","Data":"0e6f0ba718f5524fedfc696c277100f2d4cb124a3164d0a6cf4fe9430ae0dda8"}
Nov 03 23:52:17.740597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.740581    6430 scope.go:122] "RemoveContainer" containerID="0e6f0ba718f5524fedfc696c277100f2d4cb124a3164d0a6cf4fe9430ae0dda8"
Nov 03 23:52:17.743076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.743045    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:17.743170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.743112    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:17.743170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:17.743149    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:18.751027 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:18.750986    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerStarted","Data":"c7a5380310883fd1301fe6f8c9c8a53a33a9817e3ab8b2022f4da876132415f1"}
Nov 03 23:52:19.673073 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:19.673037    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:19.673286 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:19.673168    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 23:52:19.673286 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:19.673222    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 23:52:20.688408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:20.687948    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 23:52:20.688408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:20.688021    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 23:52:21.672756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:21.672720    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 23:52:21.672952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:21.672773    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 23:52:22.673173 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:22.673139    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 23:52:22.673634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:22.673190    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 23:52:23.673296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:23.673253    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 23:52:23.673818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:23.673317    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 23:52:24.681382 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:24.681349    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:24.683383 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:24.683361    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:24.684908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:24.684766    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:25.117088 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.117050    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:52:25.117330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.117300    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" podUID="f8115980-ca51-4c61-9187-4f2f646e2e09" containerName="maas-api" containerID="cri-o://9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995" gracePeriod=30
Nov 03 23:52:25.143469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.143431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:52:25.396480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.396375    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:52:25.397309 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.397255    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" podUID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" containerName="maas-api" containerID="cri-o://5609a2bbe71ff562c029bb70b8d76c6e41a414b70da7687ffccdfb18c32dd27a" gracePeriod=30
Nov 03 23:52:25.494644 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.494611    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:52:25.837581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.807745    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") pod \"f8115980-ca51-4c61-9187-4f2f646e2e09\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") "
Nov 03 23:52:25.837581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.807826    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\") pod \"f8115980-ca51-4c61-9187-4f2f646e2e09\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") "
Nov 03 23:52:25.837581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.807891    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\") pod \"f8115980-ca51-4c61-9187-4f2f646e2e09\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") "
Nov 03 23:52:25.837581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.808170    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\" (UniqueName: \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\") pod \"f8115980-ca51-4c61-9187-4f2f646e2e09\" (UID: \"f8115980-ca51-4c61-9187-4f2f646e2e09\") "
Nov 03 23:52:25.837581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.825804    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca" pod "f8115980-ca51-4c61-9187-4f2f646e2e09" (UID: "f8115980-ca51-4c61-9187-4f2f646e2e09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:25.920218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.911033    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4" pod "f8115980-ca51-4c61-9187-4f2f646e2e09" (UID: "f8115980-ca51-4c61-9187-4f2f646e2e09"). InnerVolumeSpecName "kube-api-access-wfjw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:25.926681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.925785    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjw4\" (UniqueName: \"kubernetes.io/projected/f8115980-ca51-4c61-9187-4f2f646e2e09-kube-api-access-wfjw4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:25.926681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.925821    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8115980-ca51-4c61-9187-4f2f646e2e09-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:25.927527 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.926932    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs" pod "f8115980-ca51-4c61-9187-4f2f646e2e09" (UID: "f8115980-ca51-4c61-9187-4f2f646e2e09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:25.928104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.928071    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls" pod "f8115980-ca51-4c61-9187-4f2f646e2e09" (UID: "f8115980-ca51-4c61-9187-4f2f646e2e09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:25.950603 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.950566    6430 generic.go:356] "Generic (PLEG): container finished" podID="f8115980-ca51-4c61-9187-4f2f646e2e09" containerID="9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995" exitCode=0
Nov 03 23:52:25.951345 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.951034    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"
Nov 03 23:52:25.951617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.951058    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" event={"ID":"f8115980-ca51-4c61-9187-4f2f646e2e09","Type":"ContainerDied","Data":"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"}
Nov 03 23:52:25.951785 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.951771    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq" event={"ID":"f8115980-ca51-4c61-9187-4f2f646e2e09","Type":"ContainerDied","Data":"fb24275d75c336aa11154dc788949d68e001d9ccae4f63ba8ce258108382ee5e"}
Nov 03 23:52:25.951919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.951908    6430 scope.go:122] "RemoveContainer" containerID="9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"
Nov 03 23:52:25.954227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.954210    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:52:25.957268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.957249    6430 generic.go:356] "Generic (PLEG): container finished" podID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" containerID="5609a2bbe71ff562c029bb70b8d76c6e41a414b70da7687ffccdfb18c32dd27a" exitCode=0
Nov 03 23:52:25.957777 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.957748    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" event={"ID":"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f","Type":"ContainerDied","Data":"5609a2bbe71ff562c029bb70b8d76c6e41a414b70da7687ffccdfb18c32dd27a"}
Nov 03 23:52:25.969564 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.969542    6430 scope.go:122] "RemoveContainer" containerID="9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"
Nov 03 23:52:25.969899 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:25.969875    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995\": container with ID starting with 9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995 not found: ID does not exist" containerID="9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"
Nov 03 23:52:25.969991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.969911    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995"} err="failed to get container status \"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995\": rpc error: code = NotFound desc = could not find container \"9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995\": container with ID starting with 9f1327ccde2ca65b5c4b3a978c055a1597c239d6be48a0ec0a1039c34b076995 not found: ID does not exist"
Nov 03 23:52:25.969991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:25.969950    6430 scope.go:122] "RemoveContainer" containerID="5609a2bbe71ff562c029bb70b8d76c6e41a414b70da7687ffccdfb18c32dd27a"
Nov 03 23:52:26.004853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.004816    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:52:26.008639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.008615    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c20f75-78f8558c46-r66dq"]
Nov 03 23:52:26.026931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.026880    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\") pod \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") "
Nov 03 23:52:26.027117 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.027002    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\") pod \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") "
Nov 03 23:52:26.027178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.027161    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\") pod \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") "
Nov 03 23:52:26.027329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.027305    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\" (UniqueName: \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\") pod \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\" (UID: \"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f\") "
Nov 03 23:52:26.027593 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.027561    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca" pod "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" (UID: "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:26.028580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.028553    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.028728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.028712    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.028829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.028816    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8115980-ca51-4c61-9187-4f2f646e2e09-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.029708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.029685    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml" pod "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" (UID: "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f"). InnerVolumeSpecName "kube-api-access-76fml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:26.029808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.029716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls" pod "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" (UID: "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:26.029808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.029747    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs" pod "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" (UID: "d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:26.142944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.142763    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.142944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.142804    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.142944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.142823    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fml\" (UniqueName: \"kubernetes.io/projected/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f-kube-api-access-76fml\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:26.375687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.345591    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8115980-ca51-4c61-9187-4f2f646e2e09" path="/var/lib/kubelet/pods/f8115980-ca51-4c61-9187-4f2f646e2e09/volumes"
Nov 03 23:52:26.971757 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.970800    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"
Nov 03 23:52:26.971757 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.970792    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr" event={"ID":"d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f","Type":"ContainerDied","Data":"c665760d43449e85a3e6495d06787710457208271bba178a761cc72fb50225e2"}
Nov 03 23:52:26.997033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:26.996997    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:52:27.000868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:27.000841    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-beffe3-6b89fbb8dc-8pmmr"]
Nov 03 23:52:28.234448 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:28.234407    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" path="/var/lib/kubelet/pods/d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f/volumes"
Nov 03 23:52:34.794576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:34.794531    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:52:34.795056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:34.794800    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy" containerID="cri-o://0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040" gracePeriod=30
Nov 03 23:52:34.833701 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:34.833667    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:52:35.977505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:35.977473    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:52:35.978102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:35.977650    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" podUID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" containerName="maas-api" containerID="cri-o://84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa" gracePeriod=30
Nov 03 23:52:36.408818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.408783    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:52:36.529106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529004    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\") pod \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") "
Nov 03 23:52:36.529294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529121    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\" (UniqueName: \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\") pod \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") "
Nov 03 23:52:36.529294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529145    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\") pod \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") "
Nov 03 23:52:36.529294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529258    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\") pod \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\" (UID: \"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb\") "
Nov 03 23:52:36.529568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529454    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca" pod "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" (UID: "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:36.529835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.529813    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.531296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.531268    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls" pod "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" (UID: "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:36.531402 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.531352    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69" pod "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" (UID: "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb"). InnerVolumeSpecName "kube-api-access-ztv69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:36.531841 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.531813    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs" pod "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" (UID: "6a98d67e-ab8f-49fc-9224-d0a9a1744fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:36.672016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.671977    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.672016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.672019    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztv69\" (UniqueName: \"kubernetes.io/projected/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-kube-api-access-ztv69\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:36.672314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:36.672037    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:37.065257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.065226    6430 generic.go:356] "Generic (PLEG): container finished" podID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" containerID="84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa" exitCode=0
Nov 03 23:52:37.065719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.065277    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"
Nov 03 23:52:37.065719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.065312    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" event={"ID":"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb","Type":"ContainerDied","Data":"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"}
Nov 03 23:52:37.065719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.065369    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79" event={"ID":"6a98d67e-ab8f-49fc-9224-d0a9a1744fdb","Type":"ContainerDied","Data":"f224c55ccd59eb75ac41307114bc0f961120ed762af79aebdbae322de37584a7"}
Nov 03 23:52:37.065719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.065423    6430 scope.go:122] "RemoveContainer" containerID="84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"
Nov 03 23:52:37.076314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.076295    6430 scope.go:122] "RemoveContainer" containerID="84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"
Nov 03 23:52:37.076623 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:37.076601    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa\": container with ID starting with 84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa not found: ID does not exist" containerID="84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"
Nov 03 23:52:37.076682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.076636    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa"} err="failed to get container status \"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa\": rpc error: code = NotFound desc = could not find container \"84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa\": container with ID starting with 84fd0d65814d7aff3be5798389dc59a65dfc95114c4a0a2543226b71faccbdaa not found: ID does not exist"
Nov 03 23:52:37.092123 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.092086    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:52:37.095335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:37.095314    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c887c7-66c64b4558-b2x79"]
Nov 03 23:52:38.290500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:38.290459    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" path="/var/lib/kubelet/pods/6a98d67e-ab8f-49fc-9224-d0a9a1744fdb/volumes"
Nov 03 23:52:39.289324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:39.288940    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:52:39.298592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:39.296935    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" containerID="cri-o://0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8" gracePeriod=30
Nov 03 23:52:40.072763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.072699    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:52:40.124837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.124800    6430 generic.go:356] "Generic (PLEG): container finished" podID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerID="0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040" exitCode=0
Nov 03 23:52:40.125030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.124847    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerDied","Data":"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"}
Nov 03 23:52:40.125030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.124905    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d" event={"ID":"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5","Type":"ContainerDied","Data":"f718729afbf8957e0400da8780dbf73bfadd9b3547a37ed841a84d1e2d7c4602"}
Nov 03 23:52:40.125030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.124912    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"
Nov 03 23:52:40.125030 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.124931    6430 scope.go:122] "RemoveContainer" containerID="0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"
Nov 03 23:52:40.136897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.136873    6430 scope.go:122] "RemoveContainer" containerID="21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"
Nov 03 23:52:40.149071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.149040    6430 scope.go:122] "RemoveContainer" containerID="0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"
Nov 03 23:52:40.149446 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:40.149419    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040\": container with ID starting with 0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040 not found: ID does not exist" containerID="0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"
Nov 03 23:52:40.149530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.149459    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040"} err="failed to get container status \"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040\": rpc error: code = NotFound desc = could not find container \"0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040\": container with ID starting with 0979622cc16453cee4727e50b5528022567b9234f12ca3567d2b5471d3a88040 not found: ID does not exist"
Nov 03 23:52:40.149530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.149484    6430 scope.go:122] "RemoveContainer" containerID="21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"
Nov 03 23:52:40.149816 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:40.149785    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c\": container with ID starting with 21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c not found: ID does not exist" containerID="21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"
Nov 03 23:52:40.149946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.149825    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c"} err="failed to get container status \"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c\": rpc error: code = NotFound desc = could not find container \"21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c\": container with ID starting with 21e728f2d23f024ea188a3f36f1f35ca93c79c1e0d005e912ab94dafa7b52f3c not found: ID does not exist"
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.212837    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213036    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213110    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213164    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213259    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213356    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213420    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213644    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.214410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.213706    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\") pod \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\" (UID: \"ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5\") "
Nov 03 23:52:40.215039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.214796    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:40.215039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.215033    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:40.219864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.219749    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:40.219864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.219835    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:40.220271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.220217    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:40.295438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.288142    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:40.295438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.288460    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "kube-api-access-lp8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:40.295438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.290731    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:40.295438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.290920    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo" pod "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" (UID: "ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:52:40.316591 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316546    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316577    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316615    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8wd\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-kube-api-access-lp8wd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316627    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316640    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316653    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316666    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316678    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.316995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.316692    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:40.455200 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.455159    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:52:40.457721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:40.457685    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c20f75-openshift-default-5856b8d47c-cxh7d"]
Nov 03 23:52:42.140207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:42.140162    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:52:42.140630 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:42.140243    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:52:42.232993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:42.232955    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" path="/var/lib/kubelet/pods/ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5/volumes"
Nov 03 23:52:43.458706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:43.458662    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:52:43.459179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:43.458924    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" podUID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" containerName="maas-api" containerID="cri-o://ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05" gracePeriod=30
Nov 03 23:52:43.534821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:43.534784    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:52:43.535081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:43.535034    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" podUID="7753e801-ec56-435c-adf9-2e40490ee4e4" containerName="payload-processing" containerID="cri-o://9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c" gracePeriod=30
Nov 03 23:52:44.015126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.015099    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:52:44.114052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.114018    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:52:44.127245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127195    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\" (UniqueName: \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\") pod \"7753e801-ec56-435c-adf9-2e40490ee4e4\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") "
Nov 03 23:52:44.127457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127339    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\") pod \"7753e801-ec56-435c-adf9-2e40490ee4e4\" (UID: \"7753e801-ec56-435c-adf9-2e40490ee4e4\") "
Nov 03 23:52:44.127457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127414    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\") pod \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") "
Nov 03 23:52:44.127622 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127521    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\") pod \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") "
Nov 03 23:52:44.127687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127631    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\" (UniqueName: \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\") pod \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") "
Nov 03 23:52:44.127749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127686    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\") pod \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\" (UID: \"edbb5285-0ece-4cc8-80f4-ca40ba39567f\") "
Nov 03 23:52:44.128180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.127799    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume" pod "7753e801-ec56-435c-adf9-2e40490ee4e4" (UID: "7753e801-ec56-435c-adf9-2e40490ee4e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:44.128317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.128292    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753e801-ec56-435c-adf9-2e40490ee4e4-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.128843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.128814    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca" pod "edbb5285-0ece-4cc8-80f4-ca40ba39567f" (UID: "edbb5285-0ece-4cc8-80f4-ca40ba39567f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:44.130482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.130458    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs" pod "edbb5285-0ece-4cc8-80f4-ca40ba39567f" (UID: "edbb5285-0ece-4cc8-80f4-ca40ba39567f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:44.130652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.130462    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g" pod "edbb5285-0ece-4cc8-80f4-ca40ba39567f" (UID: "edbb5285-0ece-4cc8-80f4-ca40ba39567f"). InnerVolumeSpecName "kube-api-access-g4n8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:44.130771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.130576    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb" pod "7753e801-ec56-435c-adf9-2e40490ee4e4" (UID: "7753e801-ec56-435c-adf9-2e40490ee4e4"). InnerVolumeSpecName "kube-api-access-cftkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:44.131171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.131140    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls" pod "edbb5285-0ece-4cc8-80f4-ca40ba39567f" (UID: "edbb5285-0ece-4cc8-80f4-ca40ba39567f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:52:44.185086 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.185049    6430 generic.go:356] "Generic (PLEG): container finished" podID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" containerID="ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05" exitCode=0
Nov 03 23:52:44.185298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.185279    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" event={"ID":"edbb5285-0ece-4cc8-80f4-ca40ba39567f","Type":"ContainerDied","Data":"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"}
Nov 03 23:52:44.185370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.185333    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb" event={"ID":"edbb5285-0ece-4cc8-80f4-ca40ba39567f","Type":"ContainerDied","Data":"6571dd8d6ffc7443fe8eb8be73693bdff64e6d95bdd3044ae28e9b61eac564af"}
Nov 03 23:52:44.185370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.185361    6430 scope.go:122] "RemoveContainer" containerID="ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"
Nov 03 23:52:44.185547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.185522    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"
Nov 03 23:52:44.190104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.190069    6430 generic.go:356] "Generic (PLEG): container finished" podID="7753e801-ec56-435c-adf9-2e40490ee4e4" containerID="9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c" exitCode=0
Nov 03 23:52:44.190228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.190142    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" event={"ID":"7753e801-ec56-435c-adf9-2e40490ee4e4","Type":"ContainerDied","Data":"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"}
Nov 03 23:52:44.190228 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.190189    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96" event={"ID":"7753e801-ec56-435c-adf9-2e40490ee4e4","Type":"ContainerDied","Data":"ec1bd26c73f29bee35490aff26f8e4658cea69cf593c154e1f8d8e11ac3d5517"}
Nov 03 23:52:44.190340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.190264    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"
Nov 03 23:52:44.232511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.231987    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.234136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.234085    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cftkb\" (UniqueName: \"kubernetes.io/projected/7753e801-ec56-435c-adf9-2e40490ee4e4-kube-api-access-cftkb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.234642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.234601    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edbb5285-0ece-4cc8-80f4-ca40ba39567f-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.234781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.234767    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbb5285-0ece-4cc8-80f4-ca40ba39567f-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.235181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.235141    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4n8g\" (UniqueName: \"kubernetes.io/projected/edbb5285-0ece-4cc8-80f4-ca40ba39567f-kube-api-access-g4n8g\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.313437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.313109    6430 scope.go:122] "RemoveContainer" containerID="ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"
Nov 03 23:52:44.326472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.314647    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:52:44.326472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.314693    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f5cd2-667467d9f9-fk7lb"]
Nov 03 23:52:44.344698 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:44.328147    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05\": container with ID starting with ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05 not found: ID does not exist" containerID="ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"
Nov 03 23:52:44.344698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.342640    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05"} err="failed to get container status \"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05\": rpc error: code = NotFound desc = could not find container \"ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05\": container with ID starting with ac2a7c5fe406bbb9d3f7888bac0f917e652f25918b70f2b9b8fb5d2e8c794f05 not found: ID does not exist"
Nov 03 23:52:44.344698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.342679    6430 scope.go:122] "RemoveContainer" containerID="9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"
Nov 03 23:52:44.344698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.344304    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:52:44.348358 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.348331    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f5cd2-55b5bcb49d-mbd96"]
Nov 03 23:52:44.413237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.413211    6430 scope.go:122] "RemoveContainer" containerID="9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"
Nov 03 23:52:44.413631 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:44.413609    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c\": container with ID starting with 9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c not found: ID does not exist" containerID="9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"
Nov 03 23:52:44.413721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.413644    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c"} err="failed to get container status \"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c\": rpc error: code = NotFound desc = could not find container \"9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c\": container with ID starting with 9a7cc6b4a2bc3ffe9358b2e9e3d91c16c210593c518dc544a8110a1cb3212a3c not found: ID does not exist"
Nov 03 23:52:44.579604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.579581    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:52:44.750541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750452    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750526    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750575    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750617    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750646    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750668    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750693    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.750783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750721    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.751075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750774    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:44.751075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750844    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\") pod \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\" (UID: \"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82\") "
Nov 03 23:52:44.751075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.750933    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:44.751075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751048    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:44.751278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751071    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:44.751278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751159    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:44.751653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751631    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.751737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751659    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.751737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751678    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.751737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751692    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.751737 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.751705    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.753081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.753055    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:44.753187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.753117    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:52:44.753471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.753453    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:44.753546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.753477    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v" pod "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" (UID: "ccaf11d0-a759-4d46-b4fe-ff126a7e9a82"). InnerVolumeSpecName "kube-api-access-zk74v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:44.853406 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.853358    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk74v\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-kube-api-access-zk74v\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.853596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.853429    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.853596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.853442    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:44.853596 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:44.853451    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:45.034752 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.034652    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:52:45.198730 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.198694    6430 generic.go:356] "Generic (PLEG): container finished" podID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerID="0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8" exitCode=0
Nov 03 23:52:45.198896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.198749    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"
Nov 03 23:52:45.198896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.198778    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerDied","Data":"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"}
Nov 03 23:52:45.198896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.198842    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl" event={"ID":"ccaf11d0-a759-4d46-b4fe-ff126a7e9a82","Type":"ContainerDied","Data":"f669ad2bec5f159c312bee8955251603f5918316d19d3eff7a7408ac807f04c8"}
Nov 03 23:52:45.198896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.198884    6430 scope.go:122] "RemoveContainer" containerID="0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"
Nov 03 23:52:45.212645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.212613    6430 scope.go:122] "RemoveContainer" containerID="d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"
Nov 03 23:52:45.269121 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.269043    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:52:45.282294 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.282242    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-beffe3-openshift-default-85d587945-wvlkl"]
Nov 03 23:52:45.282651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.282622    6430 scope.go:122] "RemoveContainer" containerID="0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"
Nov 03 23:52:45.285977 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:45.285907    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8\": container with ID starting with 0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8 not found: ID does not exist" containerID="0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"
Nov 03 23:52:45.285977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.285940    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8"} err="failed to get container status \"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8\": rpc error: code = NotFound desc = could not find container \"0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8\": container with ID starting with 0fe2737e4bbe19acd6dd09d7fedd4cd786beafdea3fb96e62ed85a80881456f8 not found: ID does not exist"
Nov 03 23:52:45.285977 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.285959    6430 scope.go:122] "RemoveContainer" containerID="d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"
Nov 03 23:52:45.286342 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:52:45.286320    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09\": container with ID starting with d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09 not found: ID does not exist" containerID="d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"
Nov 03 23:52:45.286489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:45.286346    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09"} err="failed to get container status \"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09\": rpc error: code = NotFound desc = could not find container \"d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09\": container with ID starting with d90a408e9ecad20a5e06f8f6cbd3ace19e256a42b189825cf99229467e57fa09 not found: ID does not exist"
Nov 03 23:52:46.235680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:46.235634    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7753e801-ec56-435c-adf9-2e40490ee4e4" path="/var/lib/kubelet/pods/7753e801-ec56-435c-adf9-2e40490ee4e4/volumes"
Nov 03 23:52:46.236357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:46.236334    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" path="/var/lib/kubelet/pods/ccaf11d0-a759-4d46-b4fe-ff126a7e9a82/volumes"
Nov 03 23:52:46.236849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:46.236826    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" path="/var/lib/kubelet/pods/edbb5285-0ece-4cc8-80f4-ca40ba39567f/volumes"
Nov 03 23:52:50.370694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.370661    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 03 23:52:50.385421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.372096    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.403653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.403623    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.403903 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.403888    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.403996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.403985    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.404087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404077    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.404163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404154    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.404245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404236    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.404316 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404307    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.404414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404403    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f8115980-ca51-4c61-9187-4f2f646e2e09" containerName="maas-api"
Nov 03 23:52:50.404515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404505    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8115980-ca51-4c61-9187-4f2f646e2e09" containerName="maas-api"
Nov 03 23:52:50.404645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404632    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" containerName="maas-api"
Nov 03 23:52:50.404731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404720    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" containerName="maas-api"
Nov 03 23:52:50.404809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404800    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7753e801-ec56-435c-adf9-2e40490ee4e4" containerName="payload-processing"
Nov 03 23:52:50.404879 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404870    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753e801-ec56-435c-adf9-2e40490ee4e4" containerName="payload-processing"
Nov 03 23:52:50.404970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.404960    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e129e2dc-6051-45ee-9c70-ef61e9283038" containerName="payload-pre-processing"
Nov 03 23:52:50.405054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405043    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e129e2dc-6051-45ee-9c70-ef61e9283038" containerName="payload-pre-processing"
Nov 03 23:52:50.405148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405137    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" containerName="maas-api"
Nov 03 23:52:50.405231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405219    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" containerName="maas-api"
Nov 03 23:52:50.405319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405308    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.405412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405382    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.405496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405487    6430 cpu_manager.go:414] "containerMap: removing container" podUID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" containerName="maas-api"
Nov 03 23:52:50.409414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.405548    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" containerName="maas-api"
Nov 03 23:52:50.409756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.409733    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.409855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.409843    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.409928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.409919    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" containerName="maas-api"
Nov 03 23:52:50.413289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.413263    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" containerName="maas-api"
Nov 03 23:52:50.413446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.413433    6430 cpu_manager.go:414] "containerMap: removing container" podUID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" containerName="payload-processing"
Nov 03 23:52:50.413541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.413531    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" containerName="payload-processing"
Nov 03 23:52:50.417410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.413819    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.417601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.417582    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05e83ac-ffe4-4d2c-ae12-bdd769f07b8f" containerName="maas-api"
Nov 03 23:52:50.417715 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.417703    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda6a7cc-7654-4aa7-b731-d6a1977340a8" containerName="payload-processing"
Nov 03 23:52:50.426407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.417797    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.426661 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.426637    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.426767 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.426757    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.426840 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.426832    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.426916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.426906    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8115980-ca51-4c61-9187-4f2f646e2e09" containerName="maas-api"
Nov 03 23:52:50.427001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.426992    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f025ba1a-bdfa-4812-8e0c-1c7895147dc5" containerName="maas-api"
Nov 03 23:52:50.427072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.427064    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753e801-ec56-435c-adf9-2e40490ee4e4" containerName="payload-processing"
Nov 03 23:52:50.427151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.427143    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e129e2dc-6051-45ee-9c70-ef61e9283038" containerName="payload-pre-processing"
Nov 03 23:52:50.427231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.427222    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a98d67e-ab8f-49fc-9224-d0a9a1744fdb" containerName="maas-api"
Nov 03 23:52:50.432421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.427293    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbb5285-0ece-4cc8-80f4-ca40ba39567f" containerName="maas-api"
Nov 03 23:52:50.462141 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.462105    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 03 23:52:50.468489 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.462609    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470102    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470130    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c125e32-7920-4c23-aa7d-f9bc57baa92a" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470148    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470156    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470363    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef91e946-5ed2-4e7f-829e-e6f2fea8b5b5" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470383    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc445bf6-97e7-46d7-9ae4-16a064f6aa4f" containerName="istio-proxy"
Nov 03 23:52:50.472107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.470412    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccaf11d0-a759-4d46-b4fe-ff126a7e9a82" containerName="istio-proxy"
Nov 03 23:52:50.475380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.475072    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 03 23:52:50.475380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.475217    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.483289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.483117    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 03 23:52:50.640044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.626597    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.640226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640057    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.640226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640111    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.640226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640159    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.640431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640230    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.640431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640296    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.640431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.640329    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.650546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650504    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.650716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650577    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.650716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650630    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.650716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650695    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.650829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650743    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.650829 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650810    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.650892 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.650870    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.651024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.651005    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk47p\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.651075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.651059    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvfh\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.651119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.651104    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.651183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.651168    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.752694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.752906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752718    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.752906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752742    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.752906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752783    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.752906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752808    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.752906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.752842    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753192 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753187    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753230    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753235    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753297    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753335    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753343    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk47p\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753641    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvfh\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753675    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753762 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753762    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.753870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753787    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.753870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753830    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.754032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753889    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.754032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753956    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.754142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.754034    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.754142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.753679    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.754333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.754238    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.754486 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.754407    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.754556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.754486    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.755102 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.755075    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.755769 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.755732    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.756722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.756699    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.756855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.756832    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.757238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.757218    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.766084 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.766059    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.766659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.766638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk47p\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.766787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.766762    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvfh\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:50.768082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.768059    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.802064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.802037    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:50.811750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:50.811724    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:51.163915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.163882    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.261050    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.261148    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.261193    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.262682    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.262755    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.264171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.262802    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:51.279046 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.278959    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 03 23:52:51.358980 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.351647    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" event={"ID":"288dd9f6-9dab-4a92-aa61-51c7a0e8e237","Type":"ContainerStarted","Data":"d9f22e572df3596dcba81620c5e3d92cc578f3a0cd96f484385ce3d5f10b200a"}
Nov 03 23:52:51.365669 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:51.364625    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" event={"ID":"a21a9e85-6bb5-4036-9817-80c15e61b786","Type":"ContainerStarted","Data":"ee43b82ba3630f8c2fa4b1609a0c4a8e288b05ec493c8d30507b06737c49143f"}
Nov 03 23:52:52.113418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.093896    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:52:52.155410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.130460    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" containerID="cri-o://c7a5380310883fd1301fe6f8c9c8a53a33a9817e3ab8b2022f4da876132415f1" gracePeriod=30
Nov 03 23:52:52.499310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.497319    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" event={"ID":"288dd9f6-9dab-4a92-aa61-51c7a0e8e237","Type":"ContainerStarted","Data":"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"}
Nov 03 23:52:52.503004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.502959    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" event={"ID":"a21a9e85-6bb5-4036-9817-80c15e61b786","Type":"ContainerStarted","Data":"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"}
Nov 03 23:52:52.525118 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.525056    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podStartSLOduration=2.5250381109999998 podStartE2EDuration="2.525038111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:52.525025658 +0000 UTC m=+1388.416087595" watchObservedRunningTime="2026-11-03 23:52:52.525038111 +0000 UTC m=+1388.416100047"
Nov 03 23:52:52.555564 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.555503    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podStartSLOduration=2.555488171 podStartE2EDuration="2.555488171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:52.553227334 +0000 UTC m=+1388.444289288" watchObservedRunningTime="2026-11-03 23:52:52.555488171 +0000 UTC m=+1388.446550105"
Nov 03 23:52:52.803176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.803144    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:52:52.804549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.804529    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:52.804617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.804571    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:52.832789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.832754    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:52:52.833190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.833167    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:52.833350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:52.833324    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:53.805830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:53.805794    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:53.806465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:53.806433    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:53.815320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:53.815283    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:53.815645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:53.815615    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:54.078245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.078158    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:52:54.084219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.084181    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.094972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.094941    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93bc1f-openshift-default-dockercfg-8m8r5\""
Nov 03 23:52:54.102541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.102520    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264215    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264276    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264375    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264508    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.269419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.264570    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2lq\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.288444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.276605    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.288444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.276681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.288444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.276714    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.378691 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.378598    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.379317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.379291    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.379943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.379924    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.380515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.380493    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.381065 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.379860    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.381207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.380898    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.381334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.381033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.450229    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.450353    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.450415    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.450498    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2lq\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.422185    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.451170    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.413125    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.467526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.451481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.479038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.478998    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.494022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.493982    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.513413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.508222    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2lq\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\") pod \"e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") " pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.718469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.718357    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:54.773716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.773582    6430 patch_prober.go:29] interesting pod/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:52:54.773971 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.773941    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:52:54.802600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.802566    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:54.802811 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.802616    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:54.823548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.823438    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:54.823548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:54.823494    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:55.163818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.163529    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:52:55.173967 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.173896    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:55.174076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.173986    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:55.174076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.174030    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:52:55.759417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.750294    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerStarted","Data":"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"}
Nov 03 23:52:55.759417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.750358    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerStarted","Data":"4690640d78ac9a0b93a0d7cc43ea42fbbf886f97ea10811bde61418f5c629d75"}
Nov 03 23:52:55.789697 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.789644    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podStartSLOduration=1.7896308950000002 podStartE2EDuration="1.789630895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:52:55.787443017 +0000 UTC m=+1391.678504937" watchObservedRunningTime="2026-11-03 23:52:55.789630895 +0000 UTC m=+1391.680692913"
Nov 03 23:52:55.803350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.803320    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:55.803562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.803366    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:55.837424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.831922    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:55.837424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:55.831981    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:56.753411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.742218    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:56.763906 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.763881    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:56.764544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.764516    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:56.765494 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.765476    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:52:56.803599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.803567    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:56.803848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.803820    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:56.815423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.814736    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:56.815423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:56.814792    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:57.787899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.787865    6430 generic.go:356] "Generic (PLEG): container finished" podID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerID="c7a5380310883fd1301fe6f8c9c8a53a33a9817e3ab8b2022f4da876132415f1" exitCode=0
Nov 03 23:52:57.788420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.788374    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerDied","Data":"c7a5380310883fd1301fe6f8c9c8a53a33a9817e3ab8b2022f4da876132415f1"}
Nov 03 23:52:57.788620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.788604    6430 scope.go:122] "RemoveContainer" containerID="0e6f0ba718f5524fedfc696c277100f2d4cb124a3164d0a6cf4fe9430ae0dda8"
Nov 03 23:52:57.804496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.802967    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:57.804496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.803028    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:57.818646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.812802    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:57.818646 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.812857    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:57.867957 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.867930    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:57.902549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.902519    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.902740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.902601    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.902806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.902743    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.903002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.902946    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903107    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903155    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903180    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903200    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903284    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\") pod \"cc763f6c-15e2-4278-9c58-acccbf764cac\" (UID: \"cc763f6c-15e2-4278-9c58-acccbf764cac\") "
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903494    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903711    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.903859    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.904216    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:57.904569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.904447    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:52:57.944458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.905568    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945236    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945285    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945301    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945316    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945332    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc763f6c-15e2-4278-9c58-acccbf764cac-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:57.971916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:57.945348    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:58.023412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.005523    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:52:58.023884 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.023810    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "kube-api-access-kgfrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:52:58.059414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.055497    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy" pod "cc763f6c-15e2-4278-9c58-acccbf764cac" (UID: "cc763f6c-15e2-4278-9c58-acccbf764cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:52:58.061009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.060936    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:58.061009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.060970    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc763f6c-15e2-4278-9c58-acccbf764cac-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:58.061009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.060986    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfrq\" (UniqueName: \"kubernetes.io/projected/cc763f6c-15e2-4278-9c58-acccbf764cac-kube-api-access-kgfrq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:52:58.796434 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.796403    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"
Nov 03 23:52:58.797015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.796969    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz" event={"ID":"cc763f6c-15e2-4278-9c58-acccbf764cac","Type":"ContainerDied","Data":"caa286788f144cb6edc289358038ba8a9b6493688682cd0d65c915a8f2eec52d"}
Nov 03 23:52:58.797140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.797056    6430 scope.go:122] "RemoveContainer" containerID="c7a5380310883fd1301fe6f8c9c8a53a33a9817e3ab8b2022f4da876132415f1"
Nov 03 23:52:58.803245 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.803219    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 23:52:58.803338 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.803278    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 23:52:58.812770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.812744    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 23:52:58.812878 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.812797    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 23:52:58.848353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.848322    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:52:58.855515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:58.855487    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f5cd2-openshift-default-545c87fd9f-pwvjz"]
Nov 03 23:52:59.769413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.768974    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:52:59.770022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.769988    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.770022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.770018    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.770250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.770058    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.770250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.770069    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.770250 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.770242    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.770470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.770260    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" containerName="istio-proxy"
Nov 03 23:52:59.776340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.776314    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:52:59.788569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.788533    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:52:59.797333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.796725    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-93bc1f-dockercfg-4hsk7\""
Nov 03 23:52:59.797333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.796864    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-93bc1f\""
Nov 03 23:52:59.801546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.801518    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:52:59.839570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.839526    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:52:59.840991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.840962    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:52:59.904748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.904713    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:52:59.905061 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.905034    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwh5\" (UniqueName: \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:52:59.905271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.905251    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:52:59.905479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:52:59.905448    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq27g\" (UniqueName: \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.045420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.042504    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwh5\" (UniqueName: \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.045420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.043548    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.045420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.043732    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq27g\" (UniqueName: \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.050438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.046723    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.066788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.066046    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.066985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.048099    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.086652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.086615    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerName="main" containerID="cri-o://0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d" gracePeriod=45
Nov 03 23:53:00.100224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.100188    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": read tcp 10.134.0.2:48216->10.134.0.118:8000: read: connection reset by peer"
Nov 03 23:53:00.131451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.127827    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq27g\" (UniqueName: \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\") pod \"payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.131451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.131085    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwh5\" (UniqueName: \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\") pod \"payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.220450 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.212377    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.335871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.333708    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc763f6c-15e2-4278-9c58-acccbf764cac" path="/var/lib/kubelet/pods/cc763f6c-15e2-4278-9c58-acccbf764cac/volumes"
Nov 03 23:53:00.421719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.421682    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:00.473047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.472571    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:53:00.474258 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.474049    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:53:00.474689 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.474628    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 03 23:53:00.599790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.599523    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658590    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\" (UniqueName: \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\" (UniqueName: \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658707    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658803    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658877    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.659371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.658916    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\") pod \"10381fc2-7f4d-4da6-abdb-20cf72960130\" (UID: \"10381fc2-7f4d-4da6-abdb-20cf72960130\") "
Nov 03 23:53:00.662959 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.661063    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:00.666285 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.666260    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:00.666988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.666965    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:53:00.667212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.667187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:00.669498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.669445    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:00.669740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.669704    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:00.669853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.669824    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf" pod "10381fc2-7f4d-4da6-abdb-20cf72960130" (UID: "10381fc2-7f4d-4da6-abdb-20cf72960130"). InnerVolumeSpecName "kube-api-access-7hkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:00.671727 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:00.671699    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9c648e_ea5e_4e43_89b1_7be0f35d6af7.slice/crio-0a51895828b96bcff0025a96bed8beea80198f4b6f1fd71f84a95ecc5d640d31 WatchSource:0}: Error finding container 0a51895828b96bcff0025a96bed8beea80198f4b6f1fd71f84a95ecc5d640d31: Status 404 returned error can't find the container with id 0a51895828b96bcff0025a96bed8beea80198f4b6f1fd71f84a95ecc5d640d31
Nov 03 23:53:00.686978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.686894    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:53:00.688656 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.688586    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:53:00.689847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.689679    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 03 23:53:00.762916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.762875    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.763148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.763120    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.763148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.763150    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.763334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.763164    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hkgf\" (UniqueName: \"kubernetes.io/projected/10381fc2-7f4d-4da6-abdb-20cf72960130-kube-api-access-7hkgf\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.763334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.763179    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10381fc2-7f4d-4da6-abdb-20cf72960130-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.763334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.763192    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10381fc2-7f4d-4da6-abdb-20cf72960130-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:00.797961 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:00.797671    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850351fa_b30e_4a73_81ec_b4db437efb62.slice/crio-0db44f9c977a4ef63ebd990b8427de9f97aed056a5b0f48c19892b76a7614567 WatchSource:0}: Error finding container 0db44f9c977a4ef63ebd990b8427de9f97aed056a5b0f48c19892b76a7614567: Status 404 returned error can't find the container with id 0db44f9c977a4ef63ebd990b8427de9f97aed056a5b0f48c19892b76a7614567
Nov 03 23:53:00.798992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.798934    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:53:00.834846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.834809    6430 generic.go:356] "Generic (PLEG): container finished" podID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerID="0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d" exitCode=0
Nov 03 23:53:00.835031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.834854    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" event={"ID":"10381fc2-7f4d-4da6-abdb-20cf72960130","Type":"ContainerDied","Data":"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"}
Nov 03 23:53:00.835031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.834899    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"
Nov 03 23:53:00.835031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.834914    6430 scope.go:122] "RemoveContainer" containerID="0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"
Nov 03 23:53:00.835031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.834902    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f" event={"ID":"10381fc2-7f4d-4da6-abdb-20cf72960130","Type":"ContainerDied","Data":"3884adebb2e89cf0b9cff367c581eae974c88ef144fd3e291eee3130e0e8f64d"}
Nov 03 23:53:00.836958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.836932    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" event={"ID":"850351fa-b30e-4a73-81ec-b4db437efb62","Type":"ContainerStarted","Data":"0db44f9c977a4ef63ebd990b8427de9f97aed056a5b0f48c19892b76a7614567"}
Nov 03 23:53:00.839437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.839376    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" event={"ID":"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7","Type":"ContainerStarted","Data":"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"}
Nov 03 23:53:00.839582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.839456    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" event={"ID":"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7","Type":"ContainerStarted","Data":"0a51895828b96bcff0025a96bed8beea80198f4b6f1fd71f84a95ecc5d640d31"}
Nov 03 23:53:00.840438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.840340    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:00.847082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.847065    6430 scope.go:122] "RemoveContainer" containerID="0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"
Nov 03 23:53:00.847339 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:53:00.847320    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d\": container with ID starting with 0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d not found: ID does not exist" containerID="0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"
Nov 03 23:53:00.847461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.847347    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d"} err="failed to get container status \"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d\": rpc error: code = NotFound desc = could not find container \"0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d\": container with ID starting with 0287575abf957ebb46e606da69f9b30288c85f43330a332eca4bd04f9ba4276d not found: ID does not exist"
Nov 03 23:53:00.894555 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.894483    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" podStartSLOduration=1.894460327 podStartE2EDuration="1.894460327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:00.883260152 +0000 UTC m=+1396.774322116" watchObservedRunningTime="2026-11-03 23:53:00.894460327 +0000 UTC m=+1396.785522257"
Nov 03 23:53:00.963262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.963184    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:53:00.988128 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:00.986761    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c7e425/praxis-route-c7e425-kserve-77786c6b45-q7r6f"]
Nov 03 23:53:01.848165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:01.848126    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" event={"ID":"850351fa-b30e-4a73-81ec-b4db437efb62","Type":"ContainerStarted","Data":"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"}
Nov 03 23:53:01.848645 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:01.848255    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:01.869674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:01.869608    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" podStartSLOduration=2.869591508 podStartE2EDuration="2.869591508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:01.868100287 +0000 UTC m=+1397.759162222" watchObservedRunningTime="2026-11-03 23:53:01.869591508 +0000 UTC m=+1397.760653440"
Nov 03 23:53:02.233333 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:02.233251    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" path="/var/lib/kubelet/pods/10381fc2-7f4d-4da6-abdb-20cf72960130/volumes"
Nov 03 23:53:04.781313 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.781265    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:53:04.782446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.782369    6430 cpu_manager.go:414] "containerMap: removing container" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerName="main"
Nov 03 23:53:04.782446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.782409    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerName="main"
Nov 03 23:53:04.782706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.782686    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="10381fc2-7f4d-4da6-abdb-20cf72960130" containerName="main"
Nov 03 23:53:04.787005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.786982    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:04.796921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.796886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:53:04.798178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.797111    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-93bc1f-serving-cert\""
Nov 03 23:53:04.843659 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:53:04.843616    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:53:04.987695 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.987638    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:04.987913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.987786    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fbt\" (UniqueName: \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:04.987913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.987873    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:04.988040 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:04.987953    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.089548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.089511    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fbt\" (UniqueName: \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.089747 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.089580    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.089747 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.089636    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.089747 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.089720    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.090305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.090282    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.092006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.091986    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.092124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.092105    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.103551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.103522    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fbt\" (UniqueName: \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\") pod \"maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.107357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.107331    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:05.422169 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.422114    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:53:05.446417 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:05.443666    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb741b4d8_763d_40e1_b65c_18a9c7a4f34c.slice/crio-a4c4fdc4f16c06b197f534ba653b9a7563e67d26de5455fde66d651dd37337d6 WatchSource:0}: Error finding container a4c4fdc4f16c06b197f534ba653b9a7563e67d26de5455fde66d651dd37337d6: Status 404 returned error can't find the container with id a4c4fdc4f16c06b197f534ba653b9a7563e67d26de5455fde66d651dd37337d6
Nov 03 23:53:05.925414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:05.925218    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" event={"ID":"b741b4d8-763d-40e1-b65c-18a9c7a4f34c","Type":"ContainerStarted","Data":"a4c4fdc4f16c06b197f534ba653b9a7563e67d26de5455fde66d651dd37337d6"}
Nov 03 23:53:06.933069 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:06.933024    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" event={"ID":"b741b4d8-763d-40e1-b65c-18a9c7a4f34c","Type":"ContainerStarted","Data":"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"}
Nov 03 23:53:06.933495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:06.933185    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:06.958149 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:06.958095    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" podStartSLOduration=2.399253122 podStartE2EDuration="2.958080773s" totalImagesPullingTime="558.827651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:06.955503065 +0000 UTC m=+1402.846565000" watchObservedRunningTime="2026-11-03 23:53:06.958080773 +0000 UTC m=+1402.849142708"
Nov 03 23:53:10.886188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:10.886144    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:53:12.871618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:12.871582    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:53:12.875421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:12.874586    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:53:12.984424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:12.983634    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:53:13.616414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.603023    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.632080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.631139    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.632080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.631957    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:53:13.701559 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.701502    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.721413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.720969    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.728284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.722271    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.758560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.758498    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.881880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.881765    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:13.983621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:13.983522    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.200212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.200093    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.291053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.290992    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:14.653400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.653166    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:53:14.722698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.672499    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.769463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.758514    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.838414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.695238    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.844076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.842025    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:53:14.869413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.844866    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:14.901781 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:14.876020    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:53:15.004472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.003533    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.223417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.215809    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:53:15.278520 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:53:15.256356    6430 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s"
Nov 03 23:53:15.317217 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.292010    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:53:15.317217 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.292076    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319208    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319286    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319324    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319360    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319449    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:53:15.332446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.319491    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:53:15.479522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.479474    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.500544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.500508    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:15.728153 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.728106    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.742772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.732626    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.877455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.868599    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.877455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.870057    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.899446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.898369    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:53:15.899791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.899760    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.929488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.926539    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:15.929488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:15.926937    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.224412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.217468    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.645408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.645325    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:16.645718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.645406    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.656775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.656735    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:16.724430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.722919    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:53:16.724430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.722990    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:53:16.777412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.744573    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.777412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.744964    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:53:16.777412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.776623    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:16.875910 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.875430    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:53:16.879176 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.879130    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.952476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.930591    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:16.952476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:16.931283    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:53:17.032541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:17.032485    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:17.107376 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:17.105601    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:53:17.107376 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:17.107193    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:17.115198 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:17.115165    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:53:17.121521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:17.121413    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:23.655419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:23.654896    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:53:23.740985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:23.740935    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:53:26.912597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:26.911526    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:26.932732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:26.931498    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.115600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.115552    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:53:27.145574 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.145523    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.595416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.590542    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.674420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.673520    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.776688 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.776621    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.778234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.778150    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.930172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.929496    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.938413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.933522    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:27.938413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:27.935178    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:28.131837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.129951    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:28.131837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.130587    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:28.242099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.224290    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:28.709514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.709461    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing" containerID="cri-o://bcd710681759de6f5c21c0f6e162321f9c6a499bde5d0b1dbff86764b4008357" gracePeriod=25
Nov 03 23:53:28.757168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.742462    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing" containerID="cri-o://a64f5d5f5283b7e711cfa12cfd701feaf19738baf779c56c6f7e1a2f3ad9b36e" gracePeriod=25
Nov 03 23:53:28.814415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.813486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:28.817758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.817722    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:28.961536 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.951345    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:53:28.981940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:28.972692    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:29.065204 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:29.065169    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:29.079202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:29.079161    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:29.252472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:29.252171    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:53:29.252648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:29.252553    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:53:30.160659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.160618    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:53:30.166504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.166482    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.178188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.178159    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b692f2-openshift-default-dockercfg-zrhph\""
Nov 03 23:53:30.182451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.182423    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:53:30.186549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186525    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186585    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186619    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186701    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186726    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186799    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186831    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjn8\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186861    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.186972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.186893    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.366974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.366929    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.368188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368158    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368401    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368528    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368554    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368647    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjn8\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368709    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368742    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.369202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.368063    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.370726    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.378190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.377872    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.379105 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.379080    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.405694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.405661    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjn8\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.417994 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.417908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.441126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.441084    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.445546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.445480    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\") pod \"e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") " pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.486423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.484191    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:30.643416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.642484    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 03 23:53:30.723052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.723020    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.757410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.752963    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 03 23:53:30.932361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932317    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932377    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932542    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932635    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932709    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbt4\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932792    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932845    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.932970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932952    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:30.933007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:30.932997    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.035723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.035683    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.035958 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.035935    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036101    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036245    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036350    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036499    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbt4\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036565    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036623    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036720    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036819    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.036895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.036838    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:53:31.037276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.037060    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.037276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.037138    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.037276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.037174    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.037276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.037248    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.039210 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.039185    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.039570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.039552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.039746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.039719    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.039793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.039781    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.039828 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.039810    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.049485 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.049457    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.050001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.049980    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbt4\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.073547 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.073506    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:31.395052 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.395017    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 03 23:53:31.402225 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:31.402180    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17c35ea_4a14_480e_af95_72c14ae0d6dc.slice/crio-c8ae5bc41ee5a3de9dadd828acc8909f9d63659af9f3a4ed31fa3f09455d19c0 WatchSource:0}: Error finding container c8ae5bc41ee5a3de9dadd828acc8909f9d63659af9f3a4ed31fa3f09455d19c0: Status 404 returned error can't find the container with id c8ae5bc41ee5a3de9dadd828acc8909f9d63659af9f3a4ed31fa3f09455d19c0
Nov 03 23:53:31.405111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.405076    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.405238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.405189    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.405306 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.405241    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:31.546305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.546227    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" event={"ID":"f17c35ea-4a14-480e-af95-72c14ae0d6dc","Type":"ContainerStarted","Data":"c8ae5bc41ee5a3de9dadd828acc8909f9d63659af9f3a4ed31fa3f09455d19c0"}
Nov 03 23:53:31.549479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.548688    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerStarted","Data":"32fa172d226c9252097922766697fd02c8835919142222355fa97c09f64f34de"}
Nov 03 23:53:31.549479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.548743    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerStarted","Data":"fca5121a66e4f7549a4b945edf8248253f22fc6d5a71047a2bd0ab291602bddb"}
Nov 03 23:53:31.577475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.577369    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podStartSLOduration=1.577349963 podStartE2EDuration="1.577349963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:31.573962851 +0000 UTC m=+1427.465024788" watchObservedRunningTime="2026-11-03 23:53:31.577349963 +0000 UTC m=+1427.468411901"
Nov 03 23:53:31.836413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.831532    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 03 23:53:31.880991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.880952    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.950699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.950664    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 03 23:53:31.956745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.956707    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957011 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.956988    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.957178    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.957341    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.957511    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.957684    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.957898 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.957879    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6q7\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.958031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.958016    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:31.958184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:31.958168    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060128    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060290    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6q7\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060322    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060368    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060573    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060609    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060856 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060663    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060877    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060925    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.060965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.060775    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.061072 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.061013    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.061435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.061336    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.061435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.061169    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.061435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.061114    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.063233 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.063198    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.063523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.063503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.074275 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.074248    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.074987 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.074963    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6q7\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.282465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.273465    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:32.484890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.484867    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:32.491459 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.491431    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:32.585441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.585382    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" event={"ID":"f17c35ea-4a14-480e-af95-72c14ae0d6dc","Type":"ContainerStarted","Data":"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"}
Nov 03 23:53:32.585607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.585472    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:32.586481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.586463    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:53:32.669038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.668923    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podStartSLOduration=2.6689031979999998 podStartE2EDuration="2.668903198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:32.610865407 +0000 UTC m=+1428.501927343" watchObservedRunningTime="2026-11-03 23:53:32.668903198 +0000 UTC m=+1428.559965136"
Nov 03 23:53:32.775897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.775856    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 03 23:53:32.791265 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:32.790187    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e409ddd_715f_4bb2_8151_ab26f2705c88.slice/crio-5f71463d9c0efa71f9d1f284c53fab932d68d05d53f4e3665f080ec79da0c094 WatchSource:0}: Error finding container 5f71463d9c0efa71f9d1f284c53fab932d68d05d53f4e3665f080ec79da0c094: Status 404 returned error can't find the container with id 5f71463d9c0efa71f9d1f284c53fab932d68d05d53f4e3665f080ec79da0c094
Nov 03 23:53:32.794960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.794424    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:32.794960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.794516    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:32.794960 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:32.794559    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:33.075610 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.075573    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:33.077525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.077495    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:33.077647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.077553    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:33.647684 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.647640    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" event={"ID":"5e409ddd-715f-4bb2-8151-ab26f2705c88","Type":"ContainerStarted","Data":"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"}
Nov 03 23:53:33.655936 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.655896    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" event={"ID":"5e409ddd-715f-4bb2-8151-ab26f2705c88","Type":"ContainerStarted","Data":"5f71463d9c0efa71f9d1f284c53fab932d68d05d53f4e3665f080ec79da0c094"}
Nov 03 23:53:33.698616 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:33.698554    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podStartSLOduration=2.698532265 podStartE2EDuration="2.698532265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:33.694779031 +0000 UTC m=+1429.585840970" watchObservedRunningTime="2026-11-03 23:53:33.698532265 +0000 UTC m=+1429.589594202"
Nov 03 23:53:34.074817 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.074773    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:34.074996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.074841    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:34.285296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.285256    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:34.286500 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.286475    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:34.286631 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.286524    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:34.354471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.353944    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:53:34.354471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:34.354181    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy" containerID="cri-o://7cc91982ec84a68c692b081b7784cad19f672878aac2501bb3adfe7b10c10b4f" gracePeriod=30
Nov 03 23:53:34.889482 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:53:34.889444    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:53:35.079932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:35.079048    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:35.079932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:35.079116    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:35.284839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:35.284754    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:35.284839 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:35.284819    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:36.084578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.079146    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:53:36.085262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.084801    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:36.085262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.084854    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:36.097076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.097032    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.097708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.096939    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:53:36.097825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.097767    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:53:36.104409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.104336    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:53:36.104534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.104450    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.105277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.105256    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b692f2\""
Nov 03 23:53:36.105445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.105310    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b692f2-dockercfg-khzwh\""
Nov 03 23:53:36.447819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.447778    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:36.448087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.448052    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:36.489081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.487118    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st52z\" (UniqueName: \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.489081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.487176    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9jj\" (UniqueName: \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.489081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.487432    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.489081 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.487501    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.600430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.595128    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.600430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.595217    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.600430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.595458    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st52z\" (UniqueName: \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.600430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.595492    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9jj\" (UniqueName: \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.600430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.596973    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.604810 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.604776    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.664460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.646896    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st52z\" (UniqueName: \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\") pod \"payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:36.664460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.649248    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9jj\" (UniqueName: \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\") pod \"payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.753463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.752984    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:36.776833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:36.776794    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:37.074743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.074697    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:37.074938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.074777    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:37.154658 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.150699    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:53:37.173979 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.169928    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:53:37.285515 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.285478    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:37.285702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.285539    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:37.772741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.770942    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" event={"ID":"a3bcfd05-6636-4d74-ad5c-115994aacbf1","Type":"ContainerStarted","Data":"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"}
Nov 03 23:53:37.772741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.771013    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" event={"ID":"a3bcfd05-6636-4d74-ad5c-115994aacbf1","Type":"ContainerStarted","Data":"1dba446d8c06a8c6033f20694654f1c6ec531da5af2f96aa9c7d3e785d6586a2"}
Nov 03 23:53:37.772741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.772210    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:37.774323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.774293    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" event={"ID":"1ce535f3-6bfc-42c1-83ae-54f72ac15647","Type":"ContainerStarted","Data":"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"}
Nov 03 23:53:37.774444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.774339    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" event={"ID":"1ce535f3-6bfc-42c1-83ae-54f72ac15647","Type":"ContainerStarted","Data":"9ce5d1814f0194af05f75bca8646b6a5117e962397f5842f19b95b04cf47af66"}
Nov 03 23:53:37.775009 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.774990    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:37.794317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:37.794271    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" podStartSLOduration=2.794246173 podStartE2EDuration="2.794246173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:37.79275433 +0000 UTC m=+1433.683816266" watchObservedRunningTime="2026-11-03 23:53:37.794246173 +0000 UTC m=+1433.685308107"
Nov 03 23:53:38.181187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:38.180775    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:38.181187 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:38.180835    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:38.294414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:38.288790    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:38.294414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:38.288857    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:39.074261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.074228    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 23:53:39.074505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.074281    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 23:53:39.253403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.253331    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:53:39.265606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.265558    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 23:53:39.285038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.284864    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:39.285038 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.284951    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:39.658568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.658504    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" podStartSLOduration=3.658484255 podStartE2EDuration="3.658484255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:37.832703447 +0000 UTC m=+1433.723765364" watchObservedRunningTime="2026-11-03 23:53:39.658484255 +0000 UTC m=+1435.549546188"
Nov 03 23:53:39.666624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.666589    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:53:39.667189 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.666851    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" podUID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" containerName="maas-api" containerID="cri-o://5f8da1a02ea59bf6aecc192d6fe6b720dd7ce2e989a287218e3e70badcc239e8" gracePeriod=30
Nov 03 23:53:39.810125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.809860    6430 generic.go:356] "Generic (PLEG): container finished" podID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" containerID="5f8da1a02ea59bf6aecc192d6fe6b720dd7ce2e989a287218e3e70badcc239e8" exitCode=0
Nov 03 23:53:39.810125 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.809976    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" event={"ID":"dae26483-6b7a-4e00-9cae-7a8d25a250c7","Type":"ContainerDied","Data":"5f8da1a02ea59bf6aecc192d6fe6b720dd7ce2e989a287218e3e70badcc239e8"}
Nov 03 23:53:39.812492 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.812417    6430 generic.go:356] "Generic (PLEG): container finished" podID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerID="7cc91982ec84a68c692b081b7784cad19f672878aac2501bb3adfe7b10c10b4f" exitCode=0
Nov 03 23:53:39.812620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.812485    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" event={"ID":"5bc2b3c6-e436-48c9-ba56-19747ae764bc","Type":"ContainerDied","Data":"7cc91982ec84a68c692b081b7784cad19f672878aac2501bb3adfe7b10c10b4f"}
Nov 03 23:53:39.812620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.812546    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk" event={"ID":"5bc2b3c6-e436-48c9-ba56-19747ae764bc","Type":"ContainerDied","Data":"77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45"}
Nov 03 23:53:39.812620 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.812573    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e0c36a1ae57a4477b5c36b553633abbcca2eabb1e3d51c3589d1cb722c9d45"
Nov 03 23:53:39.833045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:39.832683    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012349    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012442    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012551    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012710    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012752    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012808    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.012872    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.038473 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.013031    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\") pod \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\" (UID: \"5bc2b3c6-e436-48c9-ba56-19747ae764bc\") "
Nov 03 23:53:40.042408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.040809    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:40.042408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.041044    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:40.042634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.042504    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:40.048535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.042705    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:40.048535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.043580    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:40.110562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.107112    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:53:40.110562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.107876    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116264    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116295    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116311    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116326    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116340    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116353    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.116367    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc2b3c6-e436-48c9-ba56-19747ae764bc-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.127443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.118760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "kube-api-access-k79jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:40.201090 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.152626    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token" pod "5bc2b3c6-e436-48c9-ba56-19747ae764bc" (UID: "5bc2b3c6-e436-48c9-ba56-19747ae764bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:40.240749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.222960    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k79jj\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-kube-api-access-k79jj\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.240749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.222999    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc2b3c6-e436-48c9-ba56-19747ae764bc-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.288202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.287026    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 23:53:40.288202 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.287086    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 23:53:40.347525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.347493    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:53:40.419775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.419726    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:40.467832 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.440987    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:40.527517 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.526852    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 03 23:53:40.541476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.541437    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\") pod \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") "
Nov 03 23:53:40.541672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.541604    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\") pod \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") "
Nov 03 23:53:40.541813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.541779    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\" (UniqueName: \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\") pod \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") "
Nov 03 23:53:40.541860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.541822    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\") pod \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\" (UID: \"dae26483-6b7a-4e00-9cae-7a8d25a250c7\") "
Nov 03 23:53:40.542911 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.542815    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca" pod "dae26483-6b7a-4e00-9cae-7a8d25a250c7" (UID: "dae26483-6b7a-4e00-9cae-7a8d25a250c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:40.546103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.546075    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls" pod "dae26483-6b7a-4e00-9cae-7a8d25a250c7" (UID: "dae26483-6b7a-4e00-9cae-7a8d25a250c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:40.546776 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.546714    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4" pod "dae26483-6b7a-4e00-9cae-7a8d25a250c7" (UID: "dae26483-6b7a-4e00-9cae-7a8d25a250c7"). InnerVolumeSpecName "kube-api-access-cbxg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:40.547111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.547087    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs" pod "dae26483-6b7a-4e00-9cae-7a8d25a250c7" (UID: "dae26483-6b7a-4e00-9cae-7a8d25a250c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:53:40.644120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.644030    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxg4\" (UniqueName: \"kubernetes.io/projected/dae26483-6b7a-4e00-9cae-7a8d25a250c7-kube-api-access-cbxg4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.644120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.644068    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae26483-6b7a-4e00-9cae-7a8d25a250c7-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.644120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.644084    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.644120 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.644098    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dae26483-6b7a-4e00-9cae-7a8d25a250c7-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:40.821787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.821752    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"
Nov 03 23:53:40.821787 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.821773    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl" event={"ID":"dae26483-6b7a-4e00-9cae-7a8d25a250c7","Type":"ContainerDied","Data":"21b1b1913762d3a8eabb0f414f23ef64b171e373b0ead262916d8bb3312d9795"}
Nov 03 23:53:40.822047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.821850    6430 scope.go:122] "RemoveContainer" containerID="5f8da1a02ea59bf6aecc192d6fe6b720dd7ce2e989a287218e3e70badcc239e8"
Nov 03 23:53:40.822108 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.822090    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"
Nov 03 23:53:40.845661 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.845631    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:53:40.851070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.851046    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-m8wgk"]
Nov 03 23:53:40.865483 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.865198    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:53:40.872759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:40.872720    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7e425-8b94ccc59-pg5nl"]
Nov 03 23:53:41.007138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.007044    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:53:41.677441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.666817    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:41.677441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.668175    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:41.677441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.673819    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 03 23:53:41.766114 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.766079    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:41.766314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.766147    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:41.829930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.829890    6430 generic.go:356] "Generic (PLEG): container finished" podID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerID="40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac" exitCode=0
Nov 03 23:53:41.830181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.829963    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerDied","Data":"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"}
Nov 03 23:53:41.830969 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.830943    6430 scope.go:122] "RemoveContainer" containerID="40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"
Nov 03 23:53:41.832861 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.832810    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:41.833015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.832894    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:41.833015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:41.832936    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:53:42.424016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.423975    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" path="/var/lib/kubelet/pods/5bc2b3c6-e436-48c9-ba56-19747ae764bc/volumes"
Nov 03 23:53:42.428624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.428593    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" path="/var/lib/kubelet/pods/dae26483-6b7a-4e00-9cae-7a8d25a250c7/volumes"
Nov 03 23:53:42.542304 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.542214    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:53:42.543562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.543532    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" containerName="maas-api"
Nov 03 23:53:42.543857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.543838    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" containerName="maas-api"
Nov 03 23:53:42.543993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.543981    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy"
Nov 03 23:53:42.544079 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.544068    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy"
Nov 03 23:53:42.544364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.544345    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae26483-6b7a-4e00-9cae-7a8d25a250c7" containerName="maas-api"
Nov 03 23:53:42.544522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.544509    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc2b3c6-e436-48c9-ba56-19747ae764bc" containerName="istio-proxy"
Nov 03 23:53:42.591276 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.569584    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:53:42.604051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.572568    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.620487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.618642    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b692f2-serving-cert\""
Nov 03 23:53:42.694569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.685658    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.694569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.685854    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4h5\" (UniqueName: \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.694569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.686067    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.694569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.686118    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.790420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.787499    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.790420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.787686    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4h5\" (UniqueName: \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.790420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.787803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.790420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.787833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:42.909318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:42.908343    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:43.064663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.064575    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4h5\" (UniqueName: \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:43.129413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.128227    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerStarted","Data":"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"}
Nov 03 23:53:43.130427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.130258    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:43.131676 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.131627    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:53:43.164188 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.164149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:43.240442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.239257    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.256835 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.130663    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:43.365126 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.363905    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:43.690471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.674505    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.723422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.722872    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:53:43.728426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.724635    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:43.728426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.724710    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:43.732415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.731484    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 23:53:43.761414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.757641    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:53:43.888413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.887504    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.888413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.887985    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:43.917412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:43.911505    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.069413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.058587    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.139421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.138641    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:53:44.143416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.141682    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.143416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.142003    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:44.437410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.412736    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:44.441430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.438232    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:44.523422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.475189    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:53:44.523422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.479726    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:53:44.689668 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.689557    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Nov 03 23:53:44.693510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.693473    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.837412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.793790    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:44.837412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.805480    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:44.837412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:44.805537    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:45.092414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.088640    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.123456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.099109    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.123456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.099564    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:53:45.146008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.139473    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.146008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.145468    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:53:45.529381 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.499566    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:53:45.837408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.837330    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:45.969370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.969330    6430 scope.go:122] "RemoveContainer" containerID="7cc91982ec84a68c692b081b7784cad19f672878aac2501bb3adfe7b10c10b4f"
Nov 03 23:53:45.982509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:45.982457    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.016565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.016514    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:53:46.165248 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.165117    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": context deadline exceeded"
Nov 03 23:53:46.167921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.167892    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:46.168175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.168142    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:46.185539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.185496    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:53:46.187269 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.187232    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:53:46.315224 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.315177    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.358460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.339206    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" podUID="bfcab315-9f9c-4214-9fd7-36811b39c65b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.362482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.362429    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.498928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.218965    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:53:46.541355 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:53:46.541314    6430 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.851s"
Nov 03 23:53:46.695419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.694772    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:53:46.695419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.695195    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:53:46.695419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.695273    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:46.695419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.695309    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:53:46.695419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.695421    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:53:46.695828 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.695454    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:53:46.739560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.739488    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.798606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.790243    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:46.798606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.790307    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:46.824419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.820609    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:46.824747 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.824693    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.863335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.862513    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:53:46.863335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.862593    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:46.902932 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:46.902860    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh" podUID="bfcab315-9f9c-4214-9fd7-36811b39c65b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.022413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.019480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:47.067525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.063512    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.152551 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.152501    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.371103 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.370991    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.458410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.455304    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.458410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.455814    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:53:47.569411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.560448    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.620418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.604253    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:53:47.817497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.780560    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:47.817497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.781295    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:53:47.916493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.612888    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:53:47.959423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:47.934543    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:48.023647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.023583    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:48.115537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.092481    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:48.115537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.104657    6430 scope.go:122] "RemoveContainer" containerID="6a388b90973cb4c686654cfb14af6c9856b3ed7f39f97594d9f0ebedec499bcd"
Nov 03 23:53:48.115537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.114480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:53:48.154512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.154465    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:48.464549 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.444653    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:48.529453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.460486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:53:48.529453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.460527    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:53:48.650422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.611773    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:48.650422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.636219    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:48.650422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.638313    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" event={"ID":"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b","Type":"ContainerStarted","Data":"c31d85ad87a37c0cd258b17e9362570663c7e683aec69606915b285fc32a1634"}
Nov 03 23:53:48.812544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.812041    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:53:48.947921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.926363    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:48.947921 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:48.926451    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:49.128579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.124083    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:49.195410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.193678    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:49.268963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.264869    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:49.296868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.270349    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:53:49.296868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.278036    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:53:49.296868 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.278596    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:53:49.340667 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:49.340634    6430 logging.go:55] [core] [Channel #516 SubChannel #517]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.102:50052", ServerName: "10.134.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.102:50052: connect: connection refused"
Nov 03 23:53:49.470867 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:53:49.470785    6430 logging.go:55] [core] [Channel #518 SubChannel #519]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.103:50052", ServerName: "10.134.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.103:50052: connect: connection refused"
Nov 03 23:53:49.553521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.553199    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:49.558946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.555650    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:53:49.562771 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.562682    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:53:49.566501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.563147    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:53:49.573625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.573600    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:53:49.577265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.577240    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:53:49.629495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.629449    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:53:49.630942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.630920    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:53:49.637498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.637467    6430 generic.go:356] "Generic (PLEG): container finished" podID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerID="a64f5d5f5283b7e711cfa12cfd701feaf19738baf779c56c6f7e1a2f3ad9b36e" exitCode=0
Nov 03 23:53:49.637768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.637743    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" event={"ID":"2b39a41c-dde8-412e-947e-ad2ec0a37477","Type":"ContainerDied","Data":"a64f5d5f5283b7e711cfa12cfd701feaf19738baf779c56c6f7e1a2f3ad9b36e"}
Nov 03 23:53:49.643460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.643437    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:49.646328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.646300    6430 generic.go:356] "Generic (PLEG): container finished" podID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerID="bcd710681759de6f5c21c0f6e162321f9c6a499bde5d0b1dbff86764b4008357" exitCode=0
Nov 03 23:53:49.646633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.646444    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" event={"ID":"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8","Type":"ContainerDied","Data":"bcd710681759de6f5c21c0f6e162321f9c6a499bde5d0b1dbff86764b4008357"}
Nov 03 23:53:49.730418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.721197    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:49.730418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.721257    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:49.804693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.804582    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" podStartSLOduration=5.359813029 podStartE2EDuration="7.804562401s" totalImagesPullingTime="2.444749372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:49.801555923 +0000 UTC m=+1445.692617859" watchObservedRunningTime="2026-11-03 23:53:49.804562401 +0000 UTC m=+1445.695624337"
Nov 03 23:53:49.854171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.853055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:53:49.854171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.853333    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:53:49.954488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.954458    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:53:49.956505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.956479    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:53:49.974458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.974429    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\") pod \"2b39a41c-dde8-412e-947e-ad2ec0a37477\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") "
Nov 03 23:53:49.974702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.974510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\" (UniqueName: \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\") pod \"2b39a41c-dde8-412e-947e-ad2ec0a37477\" (UID: \"2b39a41c-dde8-412e-947e-ad2ec0a37477\") "
Nov 03 23:53:49.974702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.974542    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\" (UniqueName: \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\") pod \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") "
Nov 03 23:53:49.974876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.974846    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume" pod "2b39a41c-dde8-412e-947e-ad2ec0a37477" (UID: "2b39a41c-dde8-412e-947e-ad2ec0a37477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:49.975365 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.975337    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\") pod \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\" (UID: \"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8\") "
Nov 03 23:53:49.975881 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.975770    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume" pod "21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" (UID: "21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:53:49.976091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.976070    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:49.976160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.976095    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b39a41c-dde8-412e-947e-ad2ec0a37477-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:49.977552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.977524    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp" pod "21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" (UID: "21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8"). InnerVolumeSpecName "kube-api-access-l5zvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:49.981412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:49.981128    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87" pod "2b39a41c-dde8-412e-947e-ad2ec0a37477" (UID: "2b39a41c-dde8-412e-947e-ad2ec0a37477"). InnerVolumeSpecName "kube-api-access-kjg87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:53:50.174472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.136089    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjg87\" (UniqueName: \"kubernetes.io/projected/2b39a41c-dde8-412e-947e-ad2ec0a37477-kube-api-access-kjg87\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:50.174472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.136140    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zvp\" (UniqueName: \"kubernetes.io/projected/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8-kube-api-access-l5zvp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:53:50.298617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.298541    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.102:50052\" within 1s: context deadline exceeded"
Nov 03 23:53:50.298858 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.298647    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:53:50.362751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.362691    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.103:50052\" within 1s: context deadline exceeded"
Nov 03 23:53:50.362970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.362824    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:53:50.655481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.655446    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" event={"ID":"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b","Type":"ContainerStarted","Data":"e25cfb71a6fc206ec0da0d2797fdc0739d906982ec692e90b260403de82b8a90"}
Nov 03 23:53:50.656950 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.656927    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"
Nov 03 23:53:50.657064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.656974    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8" event={"ID":"21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8","Type":"ContainerDied","Data":"fb5a2ec27a4fe2cbd79e745be9436bc99eb0508f748ac231ad0c87c07b248afb"}
Nov 03 23:53:50.657064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.657036    6430 scope.go:122] "RemoveContainer" containerID="bcd710681759de6f5c21c0f6e162321f9c6a499bde5d0b1dbff86764b4008357"
Nov 03 23:53:50.658902 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.658869    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd" event={"ID":"2b39a41c-dde8-412e-947e-ad2ec0a37477","Type":"ContainerDied","Data":"8688d48b649fba34aa1543f9065d4e6803863137e8dff9c54e19e809d82fafe5"}
Nov 03 23:53:50.658995 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.658943    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"
Nov 03 23:53:50.667268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.667245    6430 scope.go:122] "RemoveContainer" containerID="a64f5d5f5283b7e711cfa12cfd701feaf19738baf779c56c6f7e1a2f3ad9b36e"
Nov 03 23:53:50.681403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.681361    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:53:50.684216 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.684188    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7e425-67bf87798c-v25qd"]
Nov 03 23:53:50.696857 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.696826    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:53:50.700667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.700641    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c7e425-649fbbdccb-gzvh8"]
Nov 03 23:53:50.731420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.730822    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:50.731420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:50.730893    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:51.726413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:51.725818    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:51.726413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:51.725882    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:52.243893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:52.243859    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" path="/var/lib/kubelet/pods/21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8/volumes"
Nov 03 23:53:52.244567 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:52.244539    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" path="/var/lib/kubelet/pods/2b39a41c-dde8-412e-947e-ad2ec0a37477/volumes"
Nov 03 23:53:52.731899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:52.731854    6430 patch_prober.go:29] interesting pod/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 23:53:52.732340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:52.731924    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 23:53:53.724374 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:53.724293    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:53:53.724652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:53.724627    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:53:53.725214 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:53.725195    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:53:56.673351 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:56.673320    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:53:59.088744 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:59.088706    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:53:59.089241 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:53:59.088957    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy" containerID="cri-o://42f0ccb14f01eab83324f231ade72daf2ea35787c87f8e72bbecac58840b0d73" gracePeriod=30
Nov 03 23:54:04.787033 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:04.786981    6430 generic.go:356] "Generic (PLEG): container finished" podID="62a1413c-dbae-41d8-b921-73036dfa63db" containerID="42f0ccb14f01eab83324f231ade72daf2ea35787c87f8e72bbecac58840b0d73" exitCode=0
Nov 03 23:54:04.792901 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:04.792692    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" event={"ID":"62a1413c-dbae-41d8-b921-73036dfa63db","Type":"ContainerDied","Data":"42f0ccb14f01eab83324f231ade72daf2ea35787c87f8e72bbecac58840b0d73"}
Nov 03 23:54:05.466895 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:05.452467    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:54:05.466895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.455687    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:05.478407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.474795    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:05.502583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.500590    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:05.529471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.504673    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:54:05.602355 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.602308    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:54:05.878554 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.841089    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.907663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.907631    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.908253 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908231    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.908601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908580    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.878255    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908772    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908903    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.883066    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.909020    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.909050    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:05.909599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.909130    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\") pod \"62a1413c-dbae-41d8-b921-73036dfa63db\" (UID: \"62a1413c-dbae-41d8-b921-73036dfa63db\") "
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.026776    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908169    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.908708    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:05.928191    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.045513    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.047703    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.047735    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.047750    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.047764    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a1413c-dbae-41d8-b921-73036dfa63db-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.079887    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.092602    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.109763    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j" event={"ID":"62a1413c-dbae-41d8-b921-73036dfa63db","Type":"ContainerDied","Data":"b4977defa21f9ee222a06d3f6480e19edddee8ff26453f286bd9b1360b4815a4"}
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.109837    6430 scope.go:122] "RemoveContainer" containerID="42f0ccb14f01eab83324f231ade72daf2ea35787c87f8e72bbecac58840b0d73"
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.110017    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"
Nov 03 23:54:06.134931 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.120991    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:06.139716 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.139619    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "kube-api-access-m25sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:06.154423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.150775    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25sq\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-kube-api-access-m25sq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.154423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.150814    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.154423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.150831    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a1413c-dbae-41d8-b921-73036dfa63db-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.163428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.162471    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.196451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.193587    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:06.211428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.205556    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo" pod "62a1413c-dbae-41d8-b921-73036dfa63db" (UID: "62a1413c-dbae-41d8-b921-73036dfa63db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:54:06.263699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.262810    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a1413c-dbae-41d8-b921-73036dfa63db-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.263699 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.262846    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a1413c-dbae-41d8-b921-73036dfa63db-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:06.477799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.476531    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:06.504445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.483957    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.562447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.545658    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:06.562447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.545751    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:06.562447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.551683    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.587007 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.564293    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:06.637012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.612911    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:54:06.637012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.613660    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:06.637012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.614776    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:06.676826 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.640257    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:06.712954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.712920    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:06.713213 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.713200    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:06.735482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.734972    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:06.926457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.889681    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:54:06.980364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:06.945475    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-slm9j"]
Nov 03 23:54:08.249244 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:08.249207    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" path="/var/lib/kubelet/pods/62a1413c-dbae-41d8-b921-73036dfa63db/volumes"
Nov 03 23:54:12.243678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:12.243635    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:54:12.248731 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:12.248702    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:54:16.233775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.233738    6430 generic.go:356] "Generic (PLEG): container finished" podID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerID="32fa172d226c9252097922766697fd02c8835919142222355fa97c09f64f34de" exitCode=0
Nov 03 23:54:16.234350 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.234325    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerDied","Data":"32fa172d226c9252097922766697fd02c8835919142222355fa97c09f64f34de"}
Nov 03 23:54:16.234806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.234792    6430 scope.go:122] "RemoveContainer" containerID="32fa172d226c9252097922766697fd02c8835919142222355fa97c09f64f34de"
Nov 03 23:54:16.236354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.236329    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:16.236437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.236422    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:16.236480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.236457    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:16.423639 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.423601    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:54:16.424078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.424032    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" podUID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" containerName="maas-api" containerID="cri-o://4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865" gracePeriod=30
Nov 03 23:54:16.970952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:16.970915    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:54:17.038999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.038947    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\") pod \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") "
Nov 03 23:54:17.039196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.039032    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\") pod \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") "
Nov 03 23:54:17.039196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.039134    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\" (UniqueName: \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\") pod \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") "
Nov 03 23:54:17.039323 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.039305    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\") pod \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\" (UID: \"b741b4d8-763d-40e1-b65c-18a9c7a4f34c\") "
Nov 03 23:54:17.045447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.043415    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs" pod "b741b4d8-763d-40e1-b65c-18a9c7a4f34c" (UID: "b741b4d8-763d-40e1-b65c-18a9c7a4f34c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:17.053456 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.049080    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.055674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.055637    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca" pod "b741b4d8-763d-40e1-b65c-18a9c7a4f34c" (UID: "b741b4d8-763d-40e1-b65c-18a9c7a4f34c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:17.056594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.056561    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls" pod "b741b4d8-763d-40e1-b65c-18a9c7a4f34c" (UID: "b741b4d8-763d-40e1-b65c-18a9c7a4f34c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:17.064653 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.064617    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt" pod "b741b4d8-763d-40e1-b65c-18a9c7a4f34c" (UID: "b741b4d8-763d-40e1-b65c-18a9c7a4f34c"). InnerVolumeSpecName "kube-api-access-j7fbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:17.217522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.217257    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fbt\" (UniqueName: \"kubernetes.io/projected/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-kube-api-access-j7fbt\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.217522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.217303    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.217522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.217320    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b741b4d8-763d-40e1-b65c-18a9c7a4f34c-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.245989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.244736    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" podUID="850351fa-b30e-4a73-81ec-b4db437efb62" containerName="payload-pre-processing" containerID="cri-o://d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec" gracePeriod=25
Nov 03 23:54:17.254427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.253977    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" podUID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" containerName="payload-processing" containerID="cri-o://5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa" gracePeriod=25
Nov 03 23:54:17.262080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.261081    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerStarted","Data":"b3f4b5e213118c82a3ce45725bcbfbe2f93d65ec7f099827f16dc221fcd9d0d7"}
Nov 03 23:54:17.263796 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.263770    6430 generic.go:356] "Generic (PLEG): container finished" podID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" containerID="4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865" exitCode=0
Nov 03 23:54:17.263869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.263855    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"
Nov 03 23:54:17.263907 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.263887    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" event={"ID":"b741b4d8-763d-40e1-b65c-18a9c7a4f34c","Type":"ContainerDied","Data":"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"}
Nov 03 23:54:17.263945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.263927    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s" event={"ID":"b741b4d8-763d-40e1-b65c-18a9c7a4f34c","Type":"ContainerDied","Data":"a4c4fdc4f16c06b197f534ba653b9a7563e67d26de5455fde66d651dd37337d6"}
Nov 03 23:54:17.263982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.263957    6430 scope.go:122] "RemoveContainer" containerID="4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"
Nov 03 23:54:17.376271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.376244    6430 scope.go:122] "RemoveContainer" containerID="4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"
Nov 03 23:54:17.376748 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:17.376709    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865\": container with ID starting with 4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865 not found: ID does not exist" containerID="4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"
Nov 03 23:54:17.376872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.376747    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865"} err="failed to get container status \"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865\": rpc error: code = NotFound desc = could not find container \"4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865\": container with ID starting with 4818d7a74f8074b07f30694015402f785432c3567308a61485515833690a0865 not found: ID does not exist"
Nov 03 23:54:17.431887 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.431842    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:54:17.471158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.467560    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93bc1f-86964b6d9-kd97s"]
Nov 03 23:54:17.485791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.485765    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:17.486961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.486933    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:17.487220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.487018    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:17.584361 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.584278    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:17.681854 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.681823    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:54:17.756257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.756166    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\") pod \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") "
Nov 03 23:54:17.756452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.756441    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\" (UniqueName: \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\") pod \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\" (UID: \"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7\") "
Nov 03 23:54:17.765198 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.765161    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5" pod "7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" (UID: "7a9c648e-ea5e-4e43-89b1-7be0f35d6af7"). InnerVolumeSpecName "kube-api-access-5pwh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:17.765830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.765798    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume" pod "7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" (UID: "7a9c648e-ea5e-4e43-89b1-7be0f35d6af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:17.772533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.772505    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:54:17.858175 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\") pod \"850351fa-b30e-4a73-81ec-b4db437efb62\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") "
Nov 03 23:54:17.858381 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858276    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\" (UniqueName: \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\") pod \"850351fa-b30e-4a73-81ec-b4db437efb62\" (UID: \"850351fa-b30e-4a73-81ec-b4db437efb62\") "
Nov 03 23:54:17.858600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858579    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume" pod "850351fa-b30e-4a73-81ec-b4db437efb62" (UID: "850351fa-b30e-4a73-81ec-b4db437efb62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:17.858938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858914    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850351fa-b30e-4a73-81ec-b4db437efb62-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.859078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858941    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwh5\" (UniqueName: \"kubernetes.io/projected/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-kube-api-access-5pwh5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.859078 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.858957    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:17.860663 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.860635    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g" pod "850351fa-b30e-4a73-81ec-b4db437efb62" (UID: "850351fa-b30e-4a73-81ec-b4db437efb62"). InnerVolumeSpecName "kube-api-access-nq27g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:17.960727 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:17.960684    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq27g\" (UniqueName: \"kubernetes.io/projected/850351fa-b30e-4a73-81ec-b4db437efb62-kube-api-access-nq27g\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:18.242478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.242441    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" path="/var/lib/kubelet/pods/b741b4d8-763d-40e1-b65c-18a9c7a4f34c/volumes"
Nov 03 23:54:18.272180 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.272148    6430 generic.go:356] "Generic (PLEG): container finished" podID="850351fa-b30e-4a73-81ec-b4db437efb62" containerID="d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec" exitCode=0
Nov 03 23:54:18.272621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.272195    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"
Nov 03 23:54:18.272621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.272224    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" event={"ID":"850351fa-b30e-4a73-81ec-b4db437efb62","Type":"ContainerDied","Data":"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"}
Nov 03 23:54:18.272621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.272288    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw" event={"ID":"850351fa-b30e-4a73-81ec-b4db437efb62","Type":"ContainerDied","Data":"0db44f9c977a4ef63ebd990b8427de9f97aed056a5b0f48c19892b76a7614567"}
Nov 03 23:54:18.272621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.272319    6430 scope.go:122] "RemoveContainer" containerID="d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"
Nov 03 23:54:18.275178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.275153    6430 generic.go:356] "Generic (PLEG): container finished" podID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" containerID="5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa" exitCode=0
Nov 03 23:54:18.276172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.275198    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"
Nov 03 23:54:18.276172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.275295    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" event={"ID":"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7","Type":"ContainerDied","Data":"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"}
Nov 03 23:54:18.276172 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.275365    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8" event={"ID":"7a9c648e-ea5e-4e43-89b1-7be0f35d6af7","Type":"ContainerDied","Data":"0a51895828b96bcff0025a96bed8beea80198f4b6f1fd71f84a95ecc5d640d31"}
Nov 03 23:54:18.283199 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.283181    6430 scope.go:122] "RemoveContainer" containerID="d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"
Nov 03 23:54:18.283489 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:18.283470    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec\": container with ID starting with d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec not found: ID does not exist" containerID="d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"
Nov 03 23:54:18.283557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.283498    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec"} err="failed to get container status \"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec\": rpc error: code = NotFound desc = could not find container \"d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec\": container with ID starting with d80be57f246f669ceb2774350dcd5c925f98d2cf62df96f4ea52de3483ca18ec not found: ID does not exist"
Nov 03 23:54:18.283557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.283516    6430 scope.go:122] "RemoveContainer" containerID="5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"
Nov 03 23:54:18.292195 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.292178    6430 scope.go:122] "RemoveContainer" containerID="5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"
Nov 03 23:54:18.292467 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:18.292447    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa\": container with ID starting with 5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa not found: ID does not exist" containerID="5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"
Nov 03 23:54:18.292524 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.292477    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa"} err="failed to get container status \"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa\": rpc error: code = NotFound desc = could not find container \"5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa\": container with ID starting with 5c5386ed578613bc2164f146edcfa29c84952865599b5374b157ac4b517d7aaa not found: ID does not exist"
Nov 03 23:54:18.298011 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.297991    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:54:18.303748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.303726    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93bc1f-7f65957bc-znxdw"]
Nov 03 23:54:18.333015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.332962    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:54:18.336822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.336780    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93bc1f-68f89bd864-zwgn8"]
Nov 03 23:54:18.484985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.484952    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:18.485178 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:18.485028    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:19.485719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:19.485686    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:19.486132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:19.485776    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:20.239843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:20.239809    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" path="/var/lib/kubelet/pods/7a9c648e-ea5e-4e43-89b1-7be0f35d6af7/volumes"
Nov 03 23:54:20.240279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:20.240264    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850351fa-b30e-4a73-81ec-b4db437efb62" path="/var/lib/kubelet/pods/850351fa-b30e-4a73-81ec-b4db437efb62/volumes"
Nov 03 23:54:20.494463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:20.488515    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:20.494463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:20.488577    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:21.485749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:21.485707    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:21.485943 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:21.485770    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:22.485876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:22.485839    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 03 23:54:22.486299 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:22.485908    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 03 23:54:23.489860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:23.489771    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:23.491862 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:23.491838    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:23.808205 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:23.808167    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:54:26.023418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.022491    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.070415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.069489    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.150413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.149472    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.607419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.595352    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:54:26.642039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.641992    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.717670 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.717546    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.719029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.718993    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.735775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.735507    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:26.752525 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:26.752476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.047487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.036494    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:54:27.047487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.038259    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:54:27.083422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.073522    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.083422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.073960    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:54:27.163543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.163494    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.243196 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.218041    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:54:27.612335 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.612289    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 03 23:54:27.631537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.631116    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.716408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.661821    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.724419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.716698    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:27.735212 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.724754    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:54:27.737694 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.725412    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:27.739939 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.739873    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:27.740266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.726600    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.758004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.740604    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.771974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.771944    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:27.773281 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.773253    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:27.783832 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.783789    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:27.785660 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.785636    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:54:27.793444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.793416    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:54:27.906935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.906846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:54:27.907257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.907237    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:54:27.907452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.907436    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:27.921531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.921502    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:27.924955 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.924925    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:54:27.925267 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.925247    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:54:27.935875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.935576    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:27.935875 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:27.935634    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:30.781682 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.781644    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 03 23:54:30.782398 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782371    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" containerName="payload-processing"
Nov 03 23:54:30.782398 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782404    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" containerName="payload-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782422    6430 cpu_manager.go:414] "containerMap: removing container" podUID="850351fa-b30e-4a73-81ec-b4db437efb62" containerName="payload-pre-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782431    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850351fa-b30e-4a73-81ec-b4db437efb62" containerName="payload-pre-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782484    6430 cpu_manager.go:414] "containerMap: removing container" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782492    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782510    6430 cpu_manager.go:414] "containerMap: removing container" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782515    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782523    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" containerName="maas-api"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782528    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" containerName="maas-api"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782536    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing"
Nov 03 23:54:30.782601 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782540    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782669    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b741b4d8-763d-40e1-b65c-18a9c7a4f34c" containerName="maas-api"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782679    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b39a41c-dde8-412e-947e-ad2ec0a37477" containerName="payload-pre-processing"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782689    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9c648e-ea5e-4e43-89b1-7be0f35d6af7" containerName="payload-processing"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782695    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="850351fa-b30e-4a73-81ec-b4db437efb62" containerName="payload-pre-processing"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782707    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ee8f77-08fc-4fb3-a3d3-2adc4cc86ec8" containerName="payload-processing"
Nov 03 23:54:30.782918 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.782714    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a1413c-dbae-41d8-b921-73036dfa63db" containerName="istio-proxy"
Nov 03 23:54:30.787773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.787753    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.794512 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.794471    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4331a7-openshift-default-dockercfg-5mfgc\""
Nov 03 23:54:30.800137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.800107    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.874599    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.874734    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.874867    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.874994    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.875059    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.875102    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jkn\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.875131    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.875223    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.875465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.875249    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979331    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979433    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979492    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979588    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979643    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979677    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jkn\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979702    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979786    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.979807    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.980696    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.980942    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.981142    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.981947    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:30.983414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:30.982532    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.016407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.015980    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.016588 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.016565    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.034798 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.034670    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jkn\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.038493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.035153    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\") pod \"e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") " pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.106940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.106599    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:31.215991 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.214012    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:54:31.220407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.216872    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy" containerID="cri-o://0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336" gracePeriod=30
Nov 03 23:54:31.503417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.492017    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 03 23:54:31.530024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.526171    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:31.530024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.526271    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:31.530024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.526314    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:31.643226 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.643189    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerStarted","Data":"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"}
Nov 03 23:54:31.643372 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.643258    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerStarted","Data":"3ec5559bf1c7c3a8020f0a6f13b9aec100287edcf2787c44bb01ec2a5c7110fa"}
Nov 03 23:54:31.671614 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:31.671549    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podStartSLOduration=1.6715292659999998 podStartE2EDuration="1.671529266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:31.668760815 +0000 UTC m=+1487.559822743" watchObservedRunningTime="2026-11-03 23:54:31.671529266 +0000 UTC m=+1487.562591207"
Nov 03 23:54:32.108171 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:32.108137    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:32.113523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:32.113490    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:32.650488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:32.650460    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:32.651476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:32.651454    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:54:34.830886 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:34.830846    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:54:35.273347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.273265    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 03 23:54:35.277441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.277383    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.279764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.279738    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 03 23:54:35.283846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.283825    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.285442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.285369    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4331a7\""
Nov 03 23:54:35.288068 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.288044    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4331a7-dockercfg-sz9qw\""
Nov 03 23:54:35.292422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.292366    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 03 23:54:35.296519 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.296493    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 03 23:54:35.457428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.455495    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcgx\" (UniqueName: \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.457428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.455576    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.457428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.455741    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrf6\" (UniqueName: \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.457428 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.455934    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.568426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.567344    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrf6\" (UniqueName: \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.568426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.567574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.568426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.567741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcgx\" (UniqueName: \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.568426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.567798    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.568819 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.568758    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.578423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.576440    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.621100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.621066    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrf6\" (UniqueName: \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\") pod \"payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:35.624232 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.624204    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcgx\" (UniqueName: \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\") pod \"payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.902708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.902612    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:35.914532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:35.914502    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:36.289442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.289407    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 03 23:54:36.311414 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:54:36.311361    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004957b7_0d92_42cc_a846_876ac4dfd3cf.slice/crio-94682d717d9e821989eaf6af6597f609c7f329d65f20c5eb952ef7db6ce37ef0 WatchSource:0}: Error finding container 94682d717d9e821989eaf6af6597f609c7f329d65f20c5eb952ef7db6ce37ef0: Status 404 returned error can't find the container with id 94682d717d9e821989eaf6af6597f609c7f329d65f20c5eb952ef7db6ce37ef0
Nov 03 23:54:36.327048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.327014    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 03 23:54:36.328131 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:54:36.328107    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499d212b_e9d7_424e_92cf_07b3f681208f.slice/crio-da15a918f0dd0b35dffa2af3e896c976fbafbd88af58c88c62984f39c31a2eb6 WatchSource:0}: Error finding container da15a918f0dd0b35dffa2af3e896c976fbafbd88af58c88c62984f39c31a2eb6: Status 404 returned error can't find the container with id da15a918f0dd0b35dffa2af3e896c976fbafbd88af58c88c62984f39c31a2eb6
Nov 03 23:54:36.511686 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.511659    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:54:36.679463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.679426    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" event={"ID":"004957b7-0d92-42cc-a846-876ac4dfd3cf","Type":"ContainerStarted","Data":"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"}
Nov 03 23:54:36.679651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.679488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" event={"ID":"004957b7-0d92-42cc-a846-876ac4dfd3cf","Type":"ContainerStarted","Data":"94682d717d9e821989eaf6af6597f609c7f329d65f20c5eb952ef7db6ce37ef0"}
Nov 03 23:54:36.679651 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.679562    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:36.681150 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.681120    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" event={"ID":"499d212b-e9d7-424e-92cf-07b3f681208f","Type":"ContainerStarted","Data":"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"}
Nov 03 23:54:36.681290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.681170    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" event={"ID":"499d212b-e9d7-424e-92cf-07b3f681208f","Type":"ContainerStarted","Data":"da15a918f0dd0b35dffa2af3e896c976fbafbd88af58c88c62984f39c31a2eb6"}
Nov 03 23:54:36.681290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.681194    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:36.682790 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.682771    6430 generic.go:356] "Generic (PLEG): container finished" podID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerID="0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336" exitCode=0
Nov 03 23:54:36.682886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.682804    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerDied","Data":"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"}
Nov 03 23:54:36.682886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.682807    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"
Nov 03 23:54:36.682886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.682830    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745" event={"ID":"ee255619-76f4-4485-8890-7e2fe23bcf8a","Type":"ContainerDied","Data":"4690640d78ac9a0b93a0d7cc43ea42fbbf886f97ea10811bde61418f5c629d75"}
Nov 03 23:54:36.682886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.682851    6430 scope.go:122] "RemoveContainer" containerID="0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"
Nov 03 23:54:36.688847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.688825    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.688974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.688861    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.688974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.688909    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.688974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.688946    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689061    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689127    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:36.689142 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689140    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689165    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689222    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689298    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\") pod \"ee255619-76f4-4485-8890-7e2fe23bcf8a\" (UID: \"ee255619-76f4-4485-8890-7e2fe23bcf8a\") "
Nov 03 23:54:36.689504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689471    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:36.689629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689603    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:36.689935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689913    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.690047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689937    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.690047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.689952    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee255619-76f4-4485-8890-7e2fe23bcf8a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.690331 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.690127    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:36.690331 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.690174    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:36.691579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.691553    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "kube-api-access-nt2lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:36.691674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.691649    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:54:36.691937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.691900    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:36.692041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.692015    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token" pod "ee255619-76f4-4485-8890-7e2fe23bcf8a" (UID: "ee255619-76f4-4485-8890-7e2fe23bcf8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:36.694156 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.694141    6430 scope.go:122] "RemoveContainer" containerID="40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"
Nov 03 23:54:36.703629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.703573    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" podStartSLOduration=1.7035566100000001 podStartE2EDuration="1.70355661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:36.701591678 +0000 UTC m=+1492.592653614" watchObservedRunningTime="2026-11-03 23:54:36.70355661 +0000 UTC m=+1492.594618549"
Nov 03 23:54:36.723407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.723361    6430 scope.go:122] "RemoveContainer" containerID="0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"
Nov 03 23:54:36.723782 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:36.723761    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336\": container with ID starting with 0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336 not found: ID does not exist" containerID="0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"
Nov 03 23:54:36.723864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.723794    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336"} err="failed to get container status \"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336\": rpc error: code = NotFound desc = could not find container \"0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336\": container with ID starting with 0bf13b619a2779affdf9a141b97c6f64f11c1c640c91ab9edd479a6e082dc336 not found: ID does not exist"
Nov 03 23:54:36.723864 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.723811    6430 scope.go:122] "RemoveContainer" containerID="40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"
Nov 03 23:54:36.724080 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:36.724068    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac\": container with ID starting with 40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac not found: ID does not exist" containerID="40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"
Nov 03 23:54:36.724124 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.724086    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac"} err="failed to get container status \"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac\": rpc error: code = NotFound desc = could not find container \"40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac\": container with ID starting with 40affe21f73eb8c58c1a8a9bd36cc1241bdcc9c4761089a9886b0985f659f1ac not found: ID does not exist"
Nov 03 23:54:36.728652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.728603    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" podStartSLOduration=1.7285900060000001 podStartE2EDuration="1.728590006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:36.726980454 +0000 UTC m=+1492.618042428" watchObservedRunningTime="2026-11-03 23:54:36.728590006 +0000 UTC m=+1492.619651941"
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791308    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791342    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791355    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791368    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2lq\" (UniqueName: \"kubernetes.io/projected/ee255619-76f4-4485-8890-7e2fe23bcf8a-kube-api-access-nt2lq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791382    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:36.791441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:36.791413    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee255619-76f4-4485-8890-7e2fe23bcf8a-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:37.010962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:37.010915    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:54:37.013855 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:37.013824    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93bc1f-openshift-default-7b75b5c977-9s745"]
Nov 03 23:54:38.322531 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:38.320999    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" path="/var/lib/kubelet/pods/ee255619-76f4-4485-8890-7e2fe23bcf8a/volumes"
Nov 03 23:54:39.119924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:39.119884    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:54:39.138024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:39.137975    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:54:40.888657 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.888614    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 03 23:54:40.890031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.889997    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:40.890031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.890030    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:40.890419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.890401    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:40.896909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.896876    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:40.901557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.901527    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4331a7-serving-cert\""
Nov 03 23:54:40.901671 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.901566    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 03 23:54:40.923319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.923285    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnmk\" (UniqueName: \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:40.923510 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.923462    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:40.923674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.923653    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:40.923743 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:40.923728    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.025312 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.025274    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.025522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.025354    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.025522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.025466    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnmk\" (UniqueName: \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.025609 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.025558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.026065 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.026039    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.027874 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.027853    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.027981 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.027908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.077377 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.077341    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnmk\" (UniqueName: \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\") pod \"maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.215113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.215012    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:41.560541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.560501    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 03 23:54:41.568976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.568925    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:54:41.569261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.569231    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerName="maas-api" containerID="cri-o://e25cfb71a6fc206ec0da0d2797fdc0739d906982ec692e90b260403de82b8a90" gracePeriod=30
Nov 03 23:54:41.668419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.668356    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.138:8443/health\": dial tcp 10.134.0.138:8443: connect: connection refused"
Nov 03 23:54:41.779488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.779451    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" event={"ID":"9cd8c821-c06d-45e8-a643-4a4825404f30","Type":"ContainerStarted","Data":"4db2dff0a3799bcb3b87de434e8bbeff3c6fe9b38c06245b2b2961d72068c42f"}
Nov 03 23:54:41.781461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.781427    6430 generic.go:356] "Generic (PLEG): container finished" podID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerID="e25cfb71a6fc206ec0da0d2797fdc0739d906982ec692e90b260403de82b8a90" exitCode=0
Nov 03 23:54:41.781635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:41.781514    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" event={"ID":"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b","Type":"ContainerDied","Data":"e25cfb71a6fc206ec0da0d2797fdc0739d906982ec692e90b260403de82b8a90"}
Nov 03 23:54:42.003207 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.002736    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.105086    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\") pod \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") "
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.105144    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\") pod \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") "
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.105225    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\") pod \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") "
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.105504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\" (UniqueName: \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\") pod \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\" (UID: \"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b\") "
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.105555    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca" pod "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" (UID: "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:42.106148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.106100    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:42.114437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.109567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls" pod "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" (UID: "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:42.114437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.109602    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs" pod "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" (UID: "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 23:54:42.114437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.109581    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5" pod "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" (UID: "a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b"). InnerVolumeSpecName "kube-api-access-jj4h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:42.246745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.246590    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:42.246745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.246629    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:42.246745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.246646    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4h5\" (UniqueName: \"kubernetes.io/projected/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b-kube-api-access-jj4h5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:42.789337 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.789305    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"
Nov 03 23:54:42.789556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.789344    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7" event={"ID":"a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b","Type":"ContainerDied","Data":"c31d85ad87a37c0cd258b17e9362570663c7e683aec69606915b285fc32a1634"}
Nov 03 23:54:42.789556 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.789424    6430 scope.go:122] "RemoveContainer" containerID="e25cfb71a6fc206ec0da0d2797fdc0739d906982ec692e90b260403de82b8a90"
Nov 03 23:54:42.791220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.791185    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" event={"ID":"9cd8c821-c06d-45e8-a643-4a4825404f30","Type":"ContainerStarted","Data":"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"}
Nov 03 23:54:42.791342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.791292    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:42.820284 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.820239    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" podStartSLOduration=2.271678008 podStartE2EDuration="2.820226201s" totalImagesPullingTime="548.548193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:42.81521629 +0000 UTC m=+1498.706278229" watchObservedRunningTime="2026-11-03 23:54:42.820226201 +0000 UTC m=+1498.711288136"
Nov 03 23:54:42.831484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.831456    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:54:42.835780 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:42.835756    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b692f2-7f4b4dfb85-m28w7"]
Nov 03 23:54:44.121229 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.121187    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" podUID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" containerName="payload-processing" containerID="cri-o://f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07" gracePeriod=25
Nov 03 23:54:44.138578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.138528    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" podUID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" containerName="payload-pre-processing" containerID="cri-o://594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4" gracePeriod=25
Nov 03 23:54:44.241259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.241215    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" path="/var/lib/kubelet/pods/a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b/volumes"
Nov 03 23:54:44.451167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.451138    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:54:44.522000 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.521971    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:54:44.594163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.586113    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\") pod \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") "
Nov 03 23:54:44.594163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.586222    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\" (UniqueName: \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\") pod \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\" (UID: \"1ce535f3-6bfc-42c1-83ae-54f72ac15647\") "
Nov 03 23:54:44.635474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.611492    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z" pod "1ce535f3-6bfc-42c1-83ae-54f72ac15647" (UID: "1ce535f3-6bfc-42c1-83ae-54f72ac15647"). InnerVolumeSpecName "kube-api-access-st52z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:44.635474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.633926    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume" pod "1ce535f3-6bfc-42c1-83ae-54f72ac15647" (UID: "1ce535f3-6bfc-42c1-83ae-54f72ac15647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:44.709522 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.709491    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\" (UniqueName: \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\") pod \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") "
Nov 03 23:54:44.709724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.709543    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\") pod \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\" (UID: \"a3bcfd05-6636-4d74-ad5c-115994aacbf1\") "
Nov 03 23:54:44.709978 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.709952    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume" pod "a3bcfd05-6636-4d74-ad5c-115994aacbf1" (UID: "a3bcfd05-6636-4d74-ad5c-115994aacbf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:44.710067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.709965    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st52z\" (UniqueName: \"kubernetes.io/projected/1ce535f3-6bfc-42c1-83ae-54f72ac15647-kube-api-access-st52z\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:44.710067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.710011    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce535f3-6bfc-42c1-83ae-54f72ac15647-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:44.711938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.711815    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj" pod "a3bcfd05-6636-4d74-ad5c-115994aacbf1" (UID: "a3bcfd05-6636-4d74-ad5c-115994aacbf1"). InnerVolumeSpecName "kube-api-access-bk9jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:44.806641 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.806605    6430 generic.go:356] "Generic (PLEG): container finished" podID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" containerID="f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07" exitCode=0
Nov 03 23:54:44.806797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.806657    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"
Nov 03 23:54:44.806797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.806689    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" event={"ID":"1ce535f3-6bfc-42c1-83ae-54f72ac15647","Type":"ContainerDied","Data":"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"}
Nov 03 23:54:44.806797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.806742    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf" event={"ID":"1ce535f3-6bfc-42c1-83ae-54f72ac15647","Type":"ContainerDied","Data":"9ce5d1814f0194af05f75bca8646b6a5117e962397f5842f19b95b04cf47af66"}
Nov 03 23:54:44.806797 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.806764    6430 scope.go:122] "RemoveContainer" containerID="f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"
Nov 03 23:54:44.808063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.808039    6430 generic.go:356] "Generic (PLEG): container finished" podID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" containerID="594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4" exitCode=0
Nov 03 23:54:44.808190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.808091    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"
Nov 03 23:54:44.808190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.808124    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" event={"ID":"a3bcfd05-6636-4d74-ad5c-115994aacbf1","Type":"ContainerDied","Data":"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"}
Nov 03 23:54:44.808190 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.808185    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw" event={"ID":"a3bcfd05-6636-4d74-ad5c-115994aacbf1","Type":"ContainerDied","Data":"1dba446d8c06a8c6033f20694654f1c6ec531da5af2f96aa9c7d3e785d6586a2"}
Nov 03 23:54:44.810773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.810751    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk9jj\" (UniqueName: \"kubernetes.io/projected/a3bcfd05-6636-4d74-ad5c-115994aacbf1-kube-api-access-bk9jj\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:44.810773 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.810773    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3bcfd05-6636-4d74-ad5c-115994aacbf1-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:44.816919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.816900    6430 scope.go:122] "RemoveContainer" containerID="f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"
Nov 03 23:54:44.817203 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:44.817184    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07\": container with ID starting with f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07 not found: ID does not exist" containerID="f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"
Nov 03 23:54:44.817261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.817212    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07"} err="failed to get container status \"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07\": rpc error: code = NotFound desc = could not find container \"f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07\": container with ID starting with f000b20c67730d11c5e32cf65fb3fe14a52ac86acb35e7490fbf107730844d07 not found: ID does not exist"
Nov 03 23:54:44.817261 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.817230    6430 scope.go:122] "RemoveContainer" containerID="594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"
Nov 03 23:54:44.827442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.827376    6430 scope.go:122] "RemoveContainer" containerID="594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"
Nov 03 23:54:44.827707 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:54:44.827688    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4\": container with ID starting with 594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4 not found: ID does not exist" containerID="594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"
Nov 03 23:54:44.827774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.827715    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4"} err="failed to get container status \"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4\": rpc error: code = NotFound desc = could not find container \"594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4\": container with ID starting with 594cb8173b630edbeab1c93c906aa2b62ab8a96c04b4b20ce9bc67e5519113c4 not found: ID does not exist"
Nov 03 23:54:44.838295 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.838269    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:54:44.848652 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.848629    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b692f2-85864f479f-tr5lw"]
Nov 03 23:54:44.865565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.865542    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:54:44.869956 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:44.869935    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b692f2-6d6d7b7568-65jcf"]
Nov 03 23:54:46.232397 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:46.232352    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" path="/var/lib/kubelet/pods/1ce535f3-6bfc-42c1-83ae-54f72ac15647/volumes"
Nov 03 23:54:46.232974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:46.232957    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" path="/var/lib/kubelet/pods/a3bcfd05-6636-4d74-ad5c-115994aacbf1/volumes"
Nov 03 23:54:47.113415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.113353    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:54:47.754016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.753988    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 03 23:54:47.754533 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.754064    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 03 23:54:47.846360 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.846245    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 03 23:54:47.847036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847013    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" containerName="payload-processing"
Nov 03 23:54:47.847036 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847036    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" containerName="payload-processing"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847067    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" containerName="payload-pre-processing"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847073    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" containerName="payload-pre-processing"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847089    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerName="maas-api"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847094    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerName="maas-api"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847116    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:47.847181 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847121    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:47.847497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847224    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce535f3-6bfc-42c1-83ae-54f72ac15647" containerName="payload-processing"
Nov 03 23:54:47.847497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847244    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee255619-76f4-4485-8890-7e2fe23bcf8a" containerName="istio-proxy"
Nov 03 23:54:47.847497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847255    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bcfd05-6636-4d74-ad5c-115994aacbf1" containerName="payload-pre-processing"
Nov 03 23:54:47.847497 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.847267    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e1e80a-ad83-4fcf-b41f-0f4dda805f1b" containerName="maas-api"
Nov 03 23:54:47.853262 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.853236    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 03 23:54:47.853419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.853363    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:47.860594 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:47.860569    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-330eb5-openshift-default-dockercfg-gxd52\""
Nov 03 23:54:48.013726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013640    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.013726 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013678    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.013914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013759    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqmx\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.013914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013796    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.013914 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013825    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.014022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013922    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.014022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013961    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.014022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.013980    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.014152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.014137    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116405    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116565    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116596    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116654    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqmx\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116708    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.116745    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.121126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.121837    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.121975    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.122061    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.122167 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.122100    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.147444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.122911    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.147444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.123163    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.147444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.123754    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.147444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.126828    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.154916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.154881    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqmx\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.158418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.157249    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.172411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.160483    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\") pod \"e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") " pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.201409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.191345    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:48.891421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:48.890874    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 03 23:54:49.569420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.558501    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:49.814413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.813377    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:54:49.949570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.949311    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:49.975453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.970816    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.997829    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.998032    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.998078    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.998225    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:49.998362    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.038461 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.010091    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 03 23:54:50.122643 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.118835    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 03 23:54:50.366086 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.354058    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 03 23:54:50.531430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.527851    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:50.531430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.528154    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:50.531430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.528333    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:54:50.832899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.827524    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:50.832899 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.832298    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:54:51.085463 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:50.973112    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.134687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.013786    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.156413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.013819    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.156940 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.013875    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.157883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.157854    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:54:51.158106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.158089    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:54:51.158259 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.013899    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.158751 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.045637    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:51.162545 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.076538    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:54:51.162718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.162691    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.187548 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.013671    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:51.187920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.164819    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:51.188164 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.188144    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:51.188336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.188323    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:51.742799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.736794    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:51.742799 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.738821    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:54:51.926197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.926102    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:54:51.954721 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.952413    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" containerID="cri-o://b3f4b5e213118c82a3ce45725bcbfbe2f93d65ec7f099827f16dc221fcd9d0d7" gracePeriod=30
Nov 03 23:54:51.992063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:51.961413    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:54:52.190474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.162157    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:54:52.191421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.162505    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.191886 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.162546    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.216416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.162581    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.340469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.192464    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.665619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.661248    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerStarted","Data":"1f6ace4a056dd330d4207356a6576d0d07e495fd98880017aaa4a538580ef77a"}
Nov 03 23:54:52.665619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.661343    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:54:52.793513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.789757    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:52.793513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:52.792380    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:54:53.056942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.056899    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:54:53.057247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.057214    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.116714 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.116647    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 03 23:54:53.196557 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.196510    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:54:53.222158 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.222108    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:54:53.223442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.223378    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:54:53.354516 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.303610    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:54:53.376414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.357745    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.459673 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.459463    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.496498 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.496464    6430 patch_prober.go:29] interesting pod/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 23:54:53.496877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.496843    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 23:54:53.526101 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.526061    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerStarted","Data":"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"}
Nov 03 23:54:53.616307 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.606686    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podStartSLOduration=6.606660594 podStartE2EDuration="6.606660594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:53.598965444 +0000 UTC m=+1509.490027395" watchObservedRunningTime="2026-11-03 23:54:53.606660594 +0000 UTC m=+1509.497722530"
Nov 03 23:54:53.746763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.745893    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:54:53.746763 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.745955    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.794059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.790620    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.794059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.793151    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:53.847130 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.824522    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 23:54:53.910476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.900930    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:54:53.962423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:53.955035    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:54:54.027079 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.027047    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:54:54.074896 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.051735    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:54:54.098061 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.097930    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:54:54.101579 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.101557    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:54:54.102768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.102734    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:54:54.139812 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.139779    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:54:54.192404 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.192300    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:54.199759 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.199732    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:54.533859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.533777    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:54.534729 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:54.534707    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:54:57.194648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.194604    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 03 23:54:57.207234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.204504    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 03 23:54:57.208367 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.208341    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.219071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.219042    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-330eb5-dockercfg-k78vt\""
Nov 03 23:54:57.219540 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.219523    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-330eb5\""
Nov 03 23:54:57.240634 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.240597    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 03 23:54:57.240783 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.240650    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 03 23:54:57.241045 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.240995    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.263444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.263409    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqj7\" (UniqueName: \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.263569 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.263532    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.263806 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.263780    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.263924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.263902    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.386464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.370325    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.386464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.370613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.386464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.370671    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.386464 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.370722    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqj7\" (UniqueName: \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.444054 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.388291    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.460422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.459126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqj7\" (UniqueName: \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\") pod \"payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.509113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.502669    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.518411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.513492    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.530330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.529998    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:57.558535 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.558492    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:57.566128 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.562808    6430 generic.go:356] "Generic (PLEG): container finished" podID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerID="b3f4b5e213118c82a3ce45725bcbfbe2f93d65ec7f099827f16dc221fcd9d0d7" exitCode=0
Nov 03 23:54:57.566451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.562838    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerDied","Data":"b3f4b5e213118c82a3ce45725bcbfbe2f93d65ec7f099827f16dc221fcd9d0d7"}
Nov 03 23:54:57.567107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.567081    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh" event={"ID":"adacbff9-8b1b-46d9-92c6-fc302d35d194","Type":"ContainerDied","Data":"fca5121a66e4f7549a4b945edf8248253f22fc6d5a71047a2bd0ab291602bddb"}
Nov 03 23:54:57.568675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.568655    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca5121a66e4f7549a4b945edf8248253f22fc6d5a71047a2bd0ab291602bddb"
Nov 03 23:54:57.568822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.567290    6430 scope.go:122] "RemoveContainer" containerID="32fa172d226c9252097922766697fd02c8835919142222355fa97c09f64f34de"
Nov 03 23:54:57.593685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.593656    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:57.697963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.697763    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.697963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.697807    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.697963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.697844    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.697963 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.697878    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698080    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698131    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:57.698314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698194    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698314 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698261    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698349    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698421    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\") pod \"adacbff9-8b1b-46d9-92c6-fc302d35d194\" (UID: \"adacbff9-8b1b-46d9-92c6-fc302d35d194\") "
Nov 03 23:54:57.698562 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698439    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 23:54:57.698723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:57.698833 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698814    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:57.698985 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.698967    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:57.699266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.699244    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.699342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.699273    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.699342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.699289    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adacbff9-8b1b-46d9-92c6-fc302d35d194-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.699342 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.699324    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.699563 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.699341    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.700922 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.700898    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 23:54:57.701993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.701973    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 23:54:57.705119 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.705096    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:57.714670 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.714644    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8" pod "adacbff9-8b1b-46d9-92c6-fc302d35d194" (UID: "adacbff9-8b1b-46d9-92c6-fc302d35d194"). InnerVolumeSpecName "kube-api-access-gkjn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 23:54:57.830451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.828713    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.830451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.828741    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.830451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.828757    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjn8\" (UniqueName: \"kubernetes.io/projected/adacbff9-8b1b-46d9-92c6-fc302d35d194-kube-api-access-gkjn8\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.830451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.828772    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adacbff9-8b1b-46d9-92c6-fc302d35d194-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 23:54:57.897311 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.897279    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 03 23:54:57.898602 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:54:57.898505    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb20907_450f_42e1_bfc6_c560ea4ab7fd.slice/crio-52926a6aa2124888f2a4f8d19cc82b78f9f95cde1a782b7e88e2d7603901eaba WatchSource:0}: Error finding container 52926a6aa2124888f2a4f8d19cc82b78f9f95cde1a782b7e88e2d7603901eaba: Status 404 returned error can't find the container with id 52926a6aa2124888f2a4f8d19cc82b78f9f95cde1a782b7e88e2d7603901eaba
Nov 03 23:54:57.901087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:57.900969    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 03 23:54:57.901807 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:54:57.901777    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9561d9a3_02b4_43a6_a1a7_4cab5c54a311.slice/crio-88c4ab9b8be3c23687fca652e0f262f370c22c4c033e415d616fd9798094fbb8 WatchSource:0}: Error finding container 88c4ab9b8be3c23687fca652e0f262f370c22c4c033e415d616fd9798094fbb8: Status 404 returned error can't find the container with id 88c4ab9b8be3c23687fca652e0f262f370c22c4c033e415d616fd9798094fbb8
Nov 03 23:54:58.641413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.638717    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"
Nov 03 23:54:58.649015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.646883    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" event={"ID":"dfb20907-450f-42e1-bfc6-c560ea4ab7fd","Type":"ContainerStarted","Data":"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"}
Nov 03 23:54:58.649015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.646943    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" event={"ID":"dfb20907-450f-42e1-bfc6-c560ea4ab7fd","Type":"ContainerStarted","Data":"52926a6aa2124888f2a4f8d19cc82b78f9f95cde1a782b7e88e2d7603901eaba"}
Nov 03 23:54:58.649015 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.647711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:54:58.674474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.652646    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" event={"ID":"9561d9a3-02b4-43a6-a1a7-4cab5c54a311","Type":"ContainerStarted","Data":"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"}
Nov 03 23:54:58.674474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.652695    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" event={"ID":"9561d9a3-02b4-43a6-a1a7-4cab5c54a311","Type":"ContainerStarted","Data":"88c4ab9b8be3c23687fca652e0f262f370c22c4c033e415d616fd9798094fbb8"}
Nov 03 23:54:58.674474 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.653380    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:54:58.697947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.695361    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" podStartSLOduration=1.695341913 podStartE2EDuration="1.695341913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:58.689993178 +0000 UTC m=+1514.581055114" watchObservedRunningTime="2026-11-03 23:54:58.695341913 +0000 UTC m=+1514.586403848"
Nov 03 23:54:58.708736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.708703    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:54:58.713938 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.713911    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b692f2-openshift-default-dbb57d7df-b7hfh"]
Nov 03 23:54:58.732135 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:54:58.732090    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" podStartSLOduration=1.73207628 podStartE2EDuration="1.73207628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:54:58.730604555 +0000 UTC m=+1514.621666493" watchObservedRunningTime="2026-11-03 23:54:58.73207628 +0000 UTC m=+1514.623138215"
Nov 03 23:55:00.233552 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:00.233520    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" path="/var/lib/kubelet/pods/adacbff9-8b1b-46d9-92c6-fc302d35d194/volumes"
Nov 03 23:55:02.378298 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.378253    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 03 23:55:02.378935 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.378919    6430 cpu_manager.go:414] "containerMap: removing container" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.378982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.378937    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.378982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.378959    6430 cpu_manager.go:414] "containerMap: removing container" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.378982 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.378965    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.379096 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.379086    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.379132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.379107    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="adacbff9-8b1b-46d9-92c6-fc302d35d194" containerName="istio-proxy"
Nov 03 23:55:02.382633 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.382610    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.397831 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.397802    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-330eb5-serving-cert\""
Nov 03 23:55:02.405407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.399777    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 03 23:55:02.543458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.541888    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.543458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.542020    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.543458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.542056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.543458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.542153    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnw7\" (UniqueName: \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.643186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.643095    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.643380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.643199    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.643380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.643225    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.643380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.643291    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnw7\" (UniqueName: \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.644014 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.643984    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.645586 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.645552    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.645824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.645800    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.656871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.656842    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnw7\" (UniqueName: \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\") pod \"maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:02.726037 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:02.726005    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:03.053470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:03.050069    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 03 23:55:03.132411 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:03.115016    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc82e6b_c5d9_4863_8712_70ebc2236f8c.slice/crio-b2a53170276e939082f36d12f6cf90bf84106ebc51c51c1b978ee4ce93881267 WatchSource:0}: Error finding container b2a53170276e939082f36d12f6cf90bf84106ebc51c51c1b978ee4ce93881267: Status 404 returned error can't find the container with id b2a53170276e939082f36d12f6cf90bf84106ebc51c51c1b978ee4ce93881267
Nov 03 23:55:03.694268 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:03.694233    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" event={"ID":"4bc82e6b-c5d9-4863-8712-70ebc2236f8c","Type":"ContainerStarted","Data":"b2a53170276e939082f36d12f6cf90bf84106ebc51c51c1b978ee4ce93881267"}
Nov 03 23:55:04.701942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:04.701900    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" event={"ID":"4bc82e6b-c5d9-4863-8712-70ebc2236f8c","Type":"ContainerStarted","Data":"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"}
Nov 03 23:55:04.702438 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:04.701984    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:04.781421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:04.781323    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" podStartSLOduration=2.26804771 podStartE2EDuration="2.78130664s" totalImagesPullingTime="513.25893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:04.778073415 +0000 UTC m=+1520.669135351" watchObservedRunningTime="2026-11-03 23:55:04.78130664 +0000 UTC m=+1520.672368571"
Nov 03 23:55:04.836458 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:04.836377    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:55:07.069528 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.069490    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 03 23:55:07.074423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.074353    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.081499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.081472    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-58b75d-openshift-default-dockercfg-ks9wt\""
Nov 03 23:55:07.085422 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.085363    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 03 23:55:07.146199 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146157    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcgj\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146278    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146453    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146476    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146594    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146647 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146614    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146702 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146652    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146746    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.146849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.146832    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270066    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270123    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270197    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270246    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270277    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270371    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcgj\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270450    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270578    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.271443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.270608    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.280088    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.280904    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.281146    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.290863    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.292452    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.307421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.305043    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.334458 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.321988    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.344412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.342234    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.363414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.362941    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcgj\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.398051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.397676    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:07.738723 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.738638    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 03 23:55:07.742377 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:07.742345    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0a15b_88da_482e_b10e_d95c47dc92bb.slice/crio-b20ef148b77f1f23fe7fad9ab5faca08784c57bd97cd12c3501077dbe773c202 WatchSource:0}: Error finding container b20ef148b77f1f23fe7fad9ab5faca08784c57bd97cd12c3501077dbe773c202: Status 404 returned error can't find the container with id b20ef148b77f1f23fe7fad9ab5faca08784c57bd97cd12c3501077dbe773c202
Nov 03 23:55:07.745032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.745002    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:07.745094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.745067    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:07.745131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:07.745099    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:08.810591 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:08.810553    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerStarted","Data":"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"}
Nov 03 23:55:08.811075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:08.810644    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerStarted","Data":"b20ef148b77f1f23fe7fad9ab5faca08784c57bd97cd12c3501077dbe773c202"}
Nov 03 23:55:08.841041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:08.840985    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podStartSLOduration=1.840968378 podStartE2EDuration="1.840968378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:08.838471734 +0000 UTC m=+1524.729533689" watchObservedRunningTime="2026-11-03 23:55:08.840968378 +0000 UTC m=+1524.732030313"
Nov 03 23:55:09.424923 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:09.419138    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:09.473613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:09.473570    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:09.817071 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:09.817039    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:09.818084 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:09.818062    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:10.006816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.006780    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 03 23:55:10.011353 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.011328    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.027815 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.027792    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-398158-openshift-default-dockercfg-zqthx\""
Nov 03 23:55:10.039168 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.039137    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 03 23:55:10.057080 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057040    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057097    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057141    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057265 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057250    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057331    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057382    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057499 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057485    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057668 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057552    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv2j\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.057668 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.057579    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175419    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175539    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175594    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175680    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175748    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv2j\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175775    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175838    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.175921    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.176464    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.185493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.176764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.208410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.187003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.211278 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.188194    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.211543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.196072    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.211655 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.196676    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.211756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.206172    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv2j\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.211888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.211181    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.233443 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.232999    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\") pod \"e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") " pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.331418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.328661    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:10.729435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.728482    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 03 23:55:10.747465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.730024    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 03 23:55:10.776706 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:10.776675    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 03 23:55:11.875416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:11.814250    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 03 23:55:11.875416 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:11.850112    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fee7e4_0040_404c_8305_fd7cbbb1c5cd.slice/crio-8e97e5425bb40079848192d6083939eb37e2a572340c8a939717d486dab95516 WatchSource:0}: Error finding container 8e97e5425bb40079848192d6083939eb37e2a572340c8a939717d486dab95516: Status 404 returned error can't find the container with id 8e97e5425bb40079848192d6083939eb37e2a572340c8a939717d486dab95516
Nov 03 23:55:11.875416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:11.867941    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:11.875416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:11.868054    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:11.875416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:11.868116    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:11.995004 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:11.994663    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.328010 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.320554    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.361151 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.361101    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.380539 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.380486    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.447755 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.447718    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerStarted","Data":"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"}
Nov 03 23:55:12.448041 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.448016    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerStarted","Data":"8e97e5425bb40079848192d6083939eb37e2a572340c8a939717d486dab95516"}
Nov 03 23:55:12.754416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.676742    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podStartSLOduration=3.676715622 podStartE2EDuration="3.676715622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:12.495972815 +0000 UTC m=+1528.387034748" watchObservedRunningTime="2026-11-03 23:55:12.676715622 +0000 UTC m=+1528.567777559"
Nov 03 23:55:12.828412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.795152    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:12.999336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:12.999277    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.003928 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.003899    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:55:13.060511 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.060468    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.088635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.088587    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.282412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.263250    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:55:13.305558 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.305506    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.349560 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.336223    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.377508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.350765    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:13.411696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.375493    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.412430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.380905    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.479328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.479275    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.479877 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.479850    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:55:13.516414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.503613    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:55:13.664503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.575538    6430 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 23:55:13.664503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.575604    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.664503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.575910    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:55:13.942470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.910107    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:13.942470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.932616    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 03 23:55:13.997415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:13.994628    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 23:55:14.207530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.204194    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:55:14.234447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.223982    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:55:14.234447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.226467    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.306521    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.322823    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjz6\" (UniqueName: \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.323083    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.350550    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd" podUID="eff13fca-8c3c-483a-92aa-b8844a6712af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.350812    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.377231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.350926    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.499605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.402993    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 03 23:55:14.500008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.499983    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.441530    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjz6\" (UniqueName: \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.506708    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.442571    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.442540    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:14.507618    6430 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-a-w2-58b75d: object "openshift-ingress"/"payload-processing-plugins-e2e-shared-a-w2-58b75d" not registered
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:14.507695    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume podName:02fea092-5031-4ce0-b79f-42501c49a619 nodeName:}" failed. No retries permitted until 2026-11-03 23:55:15.007672061 +0000 UTC m=+1530.898733999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume") pod "payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" (UID: "02fea092-5031-4ce0-b79f-42501c49a619") : object "openshift-ingress"/"payload-processing-plugins-e2e-shared-a-w2-58b75d" not registered
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.419304    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.538144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.500323    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 03 23:55:14.590415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.583883    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-58b75d-dockercfg-drvm9\""
Nov 03 23:55:14.590415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.583990    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-58b75d\""
Nov 03 23:55:14.677761 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.675189    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjz6\" (UniqueName: \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:14.849424 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.822090    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:14.860147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.822672    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:14.948412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.948324    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6rl\" (UniqueName: \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:14.948784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.948759    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:14.977165 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.977130    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 03 23:55:14.977590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.977569    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:14.977789 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.977775    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:14.977952 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.977939    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:14.978065 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:14.978189 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978178    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:14.978319 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978308    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:55:14.978481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978468    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:55:14.978612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978601    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:55:14.978738 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978726    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:55:14.978912 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:14.978900    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:55:15.000976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.000950    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:15.062357 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.062324    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:15.064847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.064446    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6rl\" (UniqueName: \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:15.064847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.064554    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:15.064847 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.064629    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:15.073029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.065756    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:15.073029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.066109    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:15.073029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.068006    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:15.115220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.115034    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6rl\" (UniqueName: \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\") pod \"payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:15.286730 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.286684    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:15.304429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.304372    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:15.885603 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.885565    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 03 23:55:15.886093 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:15.886064    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fea092_5031_4ce0_b79f_42501c49a619.slice/crio-3fe73948a1d4725a988196ddc4e154da734f93673a5984061c44d343476fdaf5 WatchSource:0}: Error finding container 3fe73948a1d4725a988196ddc4e154da734f93673a5984061c44d343476fdaf5: Status 404 returned error can't find the container with id 3fe73948a1d4725a988196ddc4e154da734f93673a5984061c44d343476fdaf5
Nov 03 23:55:15.890830 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:15.890805    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c967b0_d1e9_4e4e_a4af_b613ab839c7d.slice/crio-0d66795a0179bcc8c52568e04365bd26c7e3178e42844c465a263d14922c2f5b WatchSource:0}: Error finding container 0d66795a0179bcc8c52568e04365bd26c7e3178e42844c465a263d14922c2f5b: Status 404 returned error can't find the container with id 0d66795a0179bcc8c52568e04365bd26c7e3178e42844c465a263d14922c2f5b
Nov 03 23:55:15.900026 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:15.900000    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 03 23:55:16.262679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.262590    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 03 23:55:16.267303 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.267274    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.277348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.277310    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-398158-dockercfg-llvgz\""
Nov 03 23:55:16.277534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.277453    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-398158\""
Nov 03 23:55:16.278449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.277617    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 03 23:55:16.296067 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.296038    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 03 23:55:16.301400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.301365    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.328457 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.320431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 03 23:55:16.342051 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.342009    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.342572 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.342474    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjclp\" (UniqueName: \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.342788 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.342755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.342989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.342968    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9lj\" (UniqueName: \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.447430 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.447357    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjclp\" (UniqueName: \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.486411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.486020    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.486411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.486123    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9lj\" (UniqueName: \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.486411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.486359    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.494418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.487572    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.494418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.487572    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.535328 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.535243    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9lj\" (UniqueName: \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\") pod \"payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.548946 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.548848    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjclp\" (UniqueName: \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\") pod \"payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.597852 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.597806    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:16.671700 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.671655    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:16.702002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.701958    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" event={"ID":"41c967b0-d1e9-4e4e-a4af-b613ab839c7d","Type":"ContainerStarted","Data":"e2f546c75723787bd12243f5d09a173030eef39e9ba7261f0e8b878767a3d213"}
Nov 03 23:55:16.702138 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.702026    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" event={"ID":"41c967b0-d1e9-4e4e-a4af-b613ab839c7d","Type":"ContainerStarted","Data":"0d66795a0179bcc8c52568e04365bd26c7e3178e42844c465a263d14922c2f5b"}
Nov 03 23:55:16.703043 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.703013    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:16.726659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.725586    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" event={"ID":"02fea092-5031-4ce0-b79f-42501c49a619","Type":"ContainerStarted","Data":"02b81c304946e46f828e98c878698b79c9100e7aaeb2a8c1c57d06ca200fb5b7"}
Nov 03 23:55:16.726659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.725646    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:16.726659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.725674    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" event={"ID":"02fea092-5031-4ce0-b79f-42501c49a619","Type":"ContainerStarted","Data":"3fe73948a1d4725a988196ddc4e154da734f93673a5984061c44d343476fdaf5"}
Nov 03 23:55:16.744705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.744613    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" podStartSLOduration=3.7445991530000002 podStartE2EDuration="3.744599153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:16.737980975 +0000 UTC m=+1532.629042911" watchObservedRunningTime="2026-11-03 23:55:16.744599153 +0000 UTC m=+1532.635661088"
Nov 03 23:55:16.764681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:16.764613    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" podStartSLOduration=3.764592012 podStartE2EDuration="3.764592012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:16.760969533 +0000 UTC m=+1532.652031468" watchObservedRunningTime="2026-11-03 23:55:16.764592012 +0000 UTC m=+1532.655653949"
Nov 03 23:55:17.015986 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.003980    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 03 23:55:17.089453 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.075340    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 03 23:55:17.757592 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.757540    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" event={"ID":"f2b8ff2b-c457-417c-a7ed-76fba58bd015","Type":"ContainerStarted","Data":"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"}
Nov 03 23:55:17.757972 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.757649    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" event={"ID":"f2b8ff2b-c457-417c-a7ed-76fba58bd015","Type":"ContainerStarted","Data":"92dc1b45728fa90af6ec326d6d0ba49c4ae051af54edd6043b64290ff25014a3"}
Nov 03 23:55:17.758115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.758093    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:17.761066 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.760667    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" event={"ID":"605c8350-f8cd-4208-8185-bfcbaa38856f","Type":"ContainerStarted","Data":"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"}
Nov 03 23:55:17.761066 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.760730    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" event={"ID":"605c8350-f8cd-4208-8185-bfcbaa38856f","Type":"ContainerStarted","Data":"33b93abc5a720c165ae8e74fb96f25460e0cc15d5b8511fb251fac1a5fd9f099"}
Nov 03 23:55:17.761635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.761380    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:17.781656 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.781593    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" podStartSLOduration=1.781577513 podStartE2EDuration="1.781577513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:17.781565013 +0000 UTC m=+1533.672626962" watchObservedRunningTime="2026-11-03 23:55:17.781577513 +0000 UTC m=+1533.672639449"
Nov 03 23:55:17.806147 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:17.806072    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" podStartSLOduration=1.806051297 podStartE2EDuration="1.806051297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:17.802427161 +0000 UTC m=+1533.693489127" watchObservedRunningTime="2026-11-03 23:55:17.806051297 +0000 UTC m=+1533.697113234"
Nov 03 23:55:21.405048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.405010    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 03 23:55:21.413116 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.413052    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.420290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.420239    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-58b75d-serving-cert\""
Nov 03 23:55:21.425516 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.425480    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 03 23:55:21.450437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.450407    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z92l\" (UniqueName: \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.450613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.450586    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.450764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.450745    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.450836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.450818    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.551950 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.551913    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z92l\" (UniqueName: \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.552137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.551993    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.552137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.552078    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.552137 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.552124    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.552824 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.552796    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.554581 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.554550    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.554698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.554628    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.565193 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.565156    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z92l\" (UniqueName: \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\") pod \"maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:21.738712 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:21.738623    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:22.120411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:22.120370    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 03 23:55:22.856768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:22.856740    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" event={"ID":"0c8600c0-1c62-4155-9bb0-4a13f50bb8df","Type":"ContainerStarted","Data":"12efb13116693f72966c7ddff6991c9052fce70ed8ed6afa5fecdffda0e89938"}
Nov 03 23:55:23.212466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.212327    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 03 23:55:23.219722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.219681    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 03 23:55:23.219895 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.219845    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.225772 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.225539    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-398158-serving-cert\""
Nov 03 23:55:23.352426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.351762    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.352426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.351916    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.352426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.352142    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.352426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.352188    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.453733 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.453697    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.453948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.453759    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.453948 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:23.453869    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-serving-cert: secret "maas-api-e2e-shared-b-w3-398158-serving-cert" not found
Nov 03 23:55:23.453948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.453874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.453948 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:23.453946    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls podName:ff8bc764-815f-4618-8d83-cba9e0d57def nodeName:}" failed. No retries permitted until 2026-11-03 23:55:23.9539238 +0000 UTC m=+1539.844985732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls") pod "maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" (UID: "ff8bc764-815f-4618-8d83-cba9e0d57def") : secret "maas-api-e2e-shared-b-w3-398158-serving-cert" not found
Nov 03 23:55:23.454186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.454081    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.454948 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.454915    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.456505 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.456476    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.469897 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.469817    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.866679 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.866651    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:23.867514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.867486    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" event={"ID":"0c8600c0-1c62-4155-9bb0-4a13f50bb8df","Type":"ContainerStarted","Data":"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"}
Nov 03 23:55:23.893501 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.893451    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" podStartSLOduration=2.165060389 podStartE2EDuration="2.893422194s" totalImagesPullingTime="728.361805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:23.89056737 +0000 UTC m=+1539.781629309" watchObservedRunningTime="2026-11-03 23:55:23.893422194 +0000 UTC m=+1539.784484129"
Nov 03 23:55:23.965411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.963579    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:23.978412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:23.976945    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:24.166132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.162151    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:24.608251 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.608168    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 03 23:55:24.618927 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:24.618882    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8bc764_815f_4618_8d83_cba9e0d57def.slice/crio-af04338577a7065cec3e9dfd7b2b24053e75a436e1038d5309b775610048594b WatchSource:0}: Error finding container af04338577a7065cec3e9dfd7b2b24053e75a436e1038d5309b775610048594b: Status 404 returned error can't find the container with id af04338577a7065cec3e9dfd7b2b24053e75a436e1038d5309b775610048594b
Nov 03 23:55:24.875170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.875078    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" event={"ID":"ff8bc764-815f-4618-8d83-cba9e0d57def","Type":"ContainerStarted","Data":"af04338577a7065cec3e9dfd7b2b24053e75a436e1038d5309b775610048594b"}
Nov 03 23:55:24.876760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.876733    6430 generic.go:356] "Generic (PLEG): container finished" podID="46731542-c799-4992-89c3-221d1bd1c18a" containerID="9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576" exitCode=0
Nov 03 23:55:24.876885 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.876804    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerDied","Data":"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"}
Nov 03 23:55:24.877305 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.877290    6430 scope.go:122] "RemoveContainer" containerID="9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"
Nov 03 23:55:24.878681 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.878646    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:24.878794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.878713    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:24.878794 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:24.878746    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:25.884890 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:25.884846    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" event={"ID":"ff8bc764-815f-4618-8d83-cba9e0d57def","Type":"ContainerStarted","Data":"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"}
Nov 03 23:55:25.885371 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:25.884944    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:25.887082 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:25.887054    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerStarted","Data":"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"}
Nov 03 23:55:25.932687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:25.921044    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" podStartSLOduration=2.468225281 podStartE2EDuration="2.921024184s" totalImagesPullingTime="452.798903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:25.918858417 +0000 UTC m=+1541.809920356" watchObservedRunningTime="2026-11-03 23:55:25.921024184 +0000 UTC m=+1541.812086123"
Nov 03 23:55:26.113423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:26.112765    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:55:26.113423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:26.112923    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:26.113423 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:26.112976    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:27.133417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:27.118879    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:27.133417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:27.118940    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:27.979412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:27.978675    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 03 23:55:28.131413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:28.131313    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:28.145418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:28.137528    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:28.772106 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:28.772069    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 03 23:55:28.772290 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:28.772144    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 03 23:55:28.772347 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:28.772327    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 03 23:55:29.114417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:29.109543    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:29.114417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:29.109617    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:30.118287 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:30.118249    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:30.118962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:30.118928    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:30.893132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:30.893099    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 03 23:55:31.116418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:31.110801    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 23:55:31.116418 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:31.110871    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 23:55:31.905908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:31.905860    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 03 23:55:32.113470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:32.113423    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:55:32.113760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:32.113738    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:55:32.114364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:32.114343    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 03 23:55:32.136920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:32.136891    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:55:33.366513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.366475    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 03 23:55:33.427235 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.424718    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.427235 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.424617    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 03 23:55:33.438109 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.438082    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-b3bea7-dockercfg-vx4s7\""
Nov 03 23:55:33.494831 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.493661    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b3bea7\""
Nov 03 23:55:33.627998 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.627902    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2n5\" (UniqueName: \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.628635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.628604    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.733619 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.733574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.734872 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.734746    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.735252 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.735231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2n5\" (UniqueName: \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:33.782541 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:33.782501    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2n5\" (UniqueName: \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\") pod \"payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:34.059705 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.059668    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:34.072846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.072805    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 03 23:55:34.091266 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.091237    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.248480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.218847    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.748002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.747954    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.787613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.787509    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.787904 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.787879    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.788063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.788049    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.788186 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.788172    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.788331 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.788315    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.788667 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.788647    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.825270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.792479    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vw5\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.904329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.875986    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 03 23:55:34.937580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.928825    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 03 23:55:34.937580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.928887    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 03 23:55:34.937580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.929003    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:34.953053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.953020    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.957412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.953759    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.957677 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.957650    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.957846 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.957827    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.958012 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.957995    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.958148 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.958129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.958306 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.958285    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.968421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.958615    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.968421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.965490    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vw5\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.968421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.960335    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.969625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.969592    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:34.986784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.986751    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.008412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.993075    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.008412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.993368    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.008412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:34.960087    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.013962 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:55:35.009377    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:55:35.013962 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.009865    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.062974 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.062473    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075300    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075411    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075477    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075586    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075620    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075709    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075777    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz52h\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075819    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.075934    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.076741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.076273    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.213426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.167090    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vw5\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.213426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.184476    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.213426 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.200541    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.259360    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz52h\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.308940    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.309270    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.309565    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.309788    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.310033    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.310326    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.310544    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.310824    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.320487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.311707    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.370183 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.370142    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.385808 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.385744    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:55:35.441748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.416116    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.441748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.429084    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz52h\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.441748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.429315    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.445029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.443025    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.581469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.562452    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.582666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.582630    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.582984 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.582962    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.607409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.588373    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.607409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.588763    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:35.607409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.602657    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 23:55:35.687496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.686636    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:35.709488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.709452    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:35.855570 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:35.846488    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:36.034677 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.034634    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 23:55:36.035024 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.035005    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:36.105225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.080097    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:36.105225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.103769    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:36.105225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.104259    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:55:36.105225 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.104816    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:36.203288 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.202635    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 23:55:36.214414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.203788    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:36.214414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.212711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:36.221417 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.219538    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:36.288216 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.287647    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:36.360324 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.358596    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:36.409415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.399016    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 03 23:55:36.409415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.399367    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:36.487179 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.486464    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:36.897945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.897888    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 03 23:55:36.959451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.940050    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:36.959451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.940147    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:36.959451 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.940190    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:36.972728 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.972652    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" event={"ID":"2e782547-9b13-46d2-8298-ce8919181256","Type":"ContainerStarted","Data":"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"}
Nov 03 23:55:36.972961 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.972888    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" event={"ID":"2e782547-9b13-46d2-8298-ce8919181256","Type":"ContainerStarted","Data":"a03c75fd09acd11514191b15a8189730f7b95bd0a994188f1083d439b53b1d3a"}
Nov 03 23:55:36.973076 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:36.972942    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:37.001056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:37.000994    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" podStartSLOduration=4.000974569 podStartE2EDuration="4.000974569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:36.996235089 +0000 UTC m=+1552.887297026" watchObservedRunningTime="2026-11-03 23:55:37.000974569 +0000 UTC m=+1552.892036504"
Nov 03 23:55:37.176583 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:37.176547    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 03 23:55:37.252060 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:55:37.241269    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a62d3d_85aa_4b79_9bcc_825453ab1333.slice/crio-7d6d7e797bc42a193062df584a2d7ea9038fae799da7ddd461cb7f49392db7af WatchSource:0}: Error finding container 7d6d7e797bc42a193062df584a2d7ea9038fae799da7ddd461cb7f49392db7af: Status 404 returned error can't find the container with id 7d6d7e797bc42a193062df584a2d7ea9038fae799da7ddd461cb7f49392db7af
Nov 03 23:55:37.257758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:37.256013    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:37.257758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:37.256106    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:37.257758 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:37.256152    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:38.462850 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.462760    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" event={"ID":"17637757-1a0b-4ab1-807f-fcdaf2b8c352","Type":"ContainerStarted","Data":"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"}
Nov 03 23:55:38.482546 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.482501    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" event={"ID":"17637757-1a0b-4ab1-807f-fcdaf2b8c352","Type":"ContainerStarted","Data":"20ee1c5262f54cb77b897a5f3169d84a8d08f07cb6fcc7e75476414d4899a897"}
Nov 03 23:55:38.482863 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.482842    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" event={"ID":"80a62d3d-85aa-4b79-9bcc-825453ab1333","Type":"ContainerStarted","Data":"7d6d7e797bc42a193062df584a2d7ea9038fae799da7ddd461cb7f49392db7af"}
Nov 03 23:55:38.482993 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.478101    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podStartSLOduration=4.478078657 podStartE2EDuration="4.478078657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:38.425550109 +0000 UTC m=+1554.316612056" watchObservedRunningTime="2026-11-03 23:55:38.478078657 +0000 UTC m=+1554.369140594"
Nov 03 23:55:38.745160 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.745063    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:38.745764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.745736    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:38.745983 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:38.745946    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:39.542416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:39.539708    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" event={"ID":"80a62d3d-85aa-4b79-9bcc-825453ab1333","Type":"ContainerStarted","Data":"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"}
Nov 03 23:55:39.644412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:39.636368    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podStartSLOduration=5.636344975 podStartE2EDuration="5.636344975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:39.591843707 +0000 UTC m=+1555.482905643" watchObservedRunningTime="2026-11-03 23:55:39.636344975 +0000 UTC m=+1555.527406911"
Nov 03 23:55:39.738883 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:39.738839    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:39.739227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:39.739193    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:40.397414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.363485    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:40.397414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.369687    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:40.411444 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.407480    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:40.458414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.455163    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:40.520412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.520341    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:40.556607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.556550    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:40.557238 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.557188    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:40.615708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.615672    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:40.752465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.748830    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:40.752465 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:40.748897    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:41.043484 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.043270    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:41.299509 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.298476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 23:55:41.369532 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.369483    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:41.370400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.370356    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:55:41.372514 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.370789    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:55:41.373063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.373047    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:41.412739 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.412672    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:41.413644 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.413607    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:41.463584 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.463537    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:41.469954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.469924    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:55:41.627475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.557757    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:41.627475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.589476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:55:41.627475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.609836    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:55:41.627475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.611534    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:41.627475 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.611573    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:41.763415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.762825    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:41.763415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:41.762891    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:42.059476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.049616    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:42.069870 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.069832    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:42.432413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.388864    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:55:42.432413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.389235    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 23:55:42.469717 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.465316    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:42.492310 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.390589    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:42.522075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.521939    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 23:55:42.616413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.613878    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:55:42.616413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.614162    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 03 23:55:42.616413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.614862    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:42.616413 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.614925    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:42.653741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.642574    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:42.730481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.729507    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:42.730481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.730137    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:55:42.872534 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.872498    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:42.872809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.872780    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:42.922502 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.918008    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 03 23:55:42.990407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:42.990309    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 03 23:55:43.028414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.001936    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 03 23:55:43.028414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.002039    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 03 23:55:43.028414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.002707    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.028414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.020360    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9a5948-serving-cert\""
Nov 03 23:55:43.086400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.084871    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.086400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.084976    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.086400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.085040    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsnh\" (UniqueName: \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.086400 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.085240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.088989 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.087408    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:43.091913 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.091808    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 03 23:55:43.248094 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.247998    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.251591 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.251556    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.266414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.251881    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsnh\" (UniqueName: \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.280419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.276763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.280419 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.277916    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.298431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.298377    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 03 23:55:43.417690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.417640    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:43.529467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.521912    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:43.529467 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.522413    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:43.541439 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.536996    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.686490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.581966    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsnh\" (UniqueName: \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.686490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.582087    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:43.686490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.582131    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:43.686490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.588327    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:43.686490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.657179    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.795713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.735538    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:43.795713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.738123    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:43.795713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.738260    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:43.795713 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.744542    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:43.941447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.916744    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 03 23:55:43.941447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:43.925923    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:44.108786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.088892    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 03 23:55:44.128410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.126229    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 03 23:55:44.145523 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.144458    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 03 23:55:44.301629 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.300704    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 03 23:55:44.313481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.311745    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 03 23:55:44.313481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.313321    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.387455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.386279    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvt6\" (UniqueName: \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.387455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.386332    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.387455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.386419    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.387455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.387233    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.387801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.387662    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-b3bea7-serving-cert\""
Nov 03 23:55:44.489455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.489107    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:44.489455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.489151    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:44.496425 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.496370    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvt6\" (UniqueName: \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.496608 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.496439    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.496608 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.496492    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.496698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.496686    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.497693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.497655    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.500291 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.499819    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.500291 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.499866    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.574279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.574209    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvt6\" (UniqueName: \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\") pod \"maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.733600 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.733575    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 23:55:44.737648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.737549    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:44.737648 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.737596    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:44.823604 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.821096    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:44.834950 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:44.834891    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 03 23:55:45.400445 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.380895    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" event={"ID":"188d9733-cd39-43f3-a917-4022645d9d2e","Type":"ContainerStarted","Data":"4d97608a6b4801c153aaf82a9c506fae096a8debf69202abe9de4d54e57bbd2f"}
Nov 03 23:55:45.536678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.506919    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:45.536678 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.506994    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:45.763718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.763441    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:45.763718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.763514    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:45.977642 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:45.977568    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 03 23:55:46.402410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.399838    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" event={"ID":"188d9733-cd39-43f3-a917-4022645d9d2e","Type":"ContainerStarted","Data":"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"}
Nov 03 23:55:46.402410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.400529    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:46.402410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.402359    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" event={"ID":"fd3d8d66-451b-45cd-a6f2-599698ce696a","Type":"ContainerStarted","Data":"6781ea604f5999192721ef946ff5c6a77ff098392913a2f66b70dbc9943d0466"}
Nov 03 23:55:46.482481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.482381    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" podStartSLOduration=4.363931983 podStartE2EDuration="5.482361189s" totalImagesPullingTime="1.118429206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:46.480953065 +0000 UTC m=+1562.372015002" watchObservedRunningTime="2026-11-03 23:55:46.482361189 +0000 UTC m=+1562.373423121"
Nov 03 23:55:46.489085 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.489051    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:46.489220 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.489108    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:46.736344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.735824    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:46.736344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:46.735888    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:47.417893 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.417850    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" event={"ID":"fd3d8d66-451b-45cd-a6f2-599698ce696a","Type":"ContainerStarted","Data":"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"}
Nov 03 23:55:47.418478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.417945    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:47.440339 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.440271    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" podStartSLOduration=2.795481272 podStartE2EDuration="3.440250628s" totalImagesPullingTime="644.769356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:47.439966714 +0000 UTC m=+1563.331028647" watchObservedRunningTime="2026-11-03 23:55:47.440250628 +0000 UTC m=+1563.331312564"
Nov 03 23:55:47.487920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.487881    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:47.488170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.487948    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:47.736098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.735999    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 23:55:47.736098 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:47.736069    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 23:55:48.490431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:48.488308    6430 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 23:55:48.490431 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:48.488413    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 23:55:48.839760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:48.797678    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:48.839760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:48.799350    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:48.839760 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:48.812840    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 03 23:55:49.493659 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:49.493615    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:49.494152 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:49.494028    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:49.494746 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:49.494723    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 03 23:55:49.542439 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:49.542372    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 03 23:55:51.467482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.466984    6430 generic.go:356] "Generic (PLEG): container finished" podID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerID="d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc" exitCode=0
Nov 03 23:55:51.467482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.467174    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerDied","Data":"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"}
Nov 03 23:55:51.471408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.468118    6430 scope.go:122] "RemoveContainer" containerID="d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"
Nov 03 23:55:51.471408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.471358    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:51.472144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.471495    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:51.472144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.471540    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:51.472144 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.471982    6430 generic.go:356] "Generic (PLEG): container finished" podID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerID="e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e" exitCode=0
Nov 03 23:55:51.473113 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.473086    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerDied","Data":"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"}
Nov 03 23:55:51.473722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.473703    6430 scope.go:122] "RemoveContainer" containerID="e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"
Nov 03 23:55:51.477674 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.477642    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:51.477876 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.477852    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:51.478414 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:51.478190    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:55:52.502576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:52.502544    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 03 23:55:52.977420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:52.961871    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerStarted","Data":"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"}
Nov 03 23:55:53.028526 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.026176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerStarted","Data":"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"}
Nov 03 23:55:53.248470 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.230453    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:55:53.273409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.258675    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:53.273409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.258736    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:53.399345 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.399305    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:53.400344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.400316    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:53.400482 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:53.400383    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:54.192091 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.192052    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:54.192543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.192106    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:54.418411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.414943    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:54.418411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.415008    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:54.483675 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.483593    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 03 23:55:54.542410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.542234    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:55:54.833409 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:54.823719    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:55:55.209493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:55.206765    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:55.209493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:55.206827    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:55.410816 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:55.410776    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:55.411099 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:55.411068    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:55.609407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:55.608652    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:55:56.203480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:56.199817    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:56.203480 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:56.199894    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:56.414764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:56.414729    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:56.415340 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:56.415306    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:56.525801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:56.525700    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:57.023565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.023509    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:57.106825 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.096476    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:57.184420 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.182356    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 23:55:57.197472 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.197435    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:57.197764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.197724    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:57.302490 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.298908    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:57.402234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.401185    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:57.402234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.401249    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:57.411460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.410660    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:57.435719 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:57.432888    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:58.302843 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:58.302805    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:58.303143 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:58.303111    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:58.431437 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:58.408279    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:58.431770 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:58.431725    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:59.126606 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.126538    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:59.207571 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.207532    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:55:59.207837 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.207811    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:55:59.404666 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.404550    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:59.405132 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.405099    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 03 23:55:59.429452 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.421428    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:55:59.429756 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.429719    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:55:59.563455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.556484    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:59.563455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.556964    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:55:59.683597 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:55:59.668337    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 23:56:00.227410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:00.221472    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 23:56:00.227410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:00.221538    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 23:56:00.227410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:00.221961    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 03 23:56:00.398786 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:00.398741    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 23:56:00.398966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:00.398812    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 23:56:01.197344 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:01.197303    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:56:01.198378 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:01.198355    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 03 23:56:01.404240 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:01.404204    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:56:01.405442 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:01.405380    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 03 23:56:04.831793 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:56:04.831754    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:56:09.436318 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.436280    6430 generic.go:356] "Generic (PLEG): container finished" podID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerID="f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475" exitCode=0
Nov 03 23:56:09.436813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.436353    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerDied","Data":"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"}
Nov 03 23:56:09.436869 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.436840    6430 scope.go:122] "RemoveContainer" containerID="f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"
Nov 03 23:56:09.438612 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.438584    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:56:09.438683 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.438656    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:56:09.438722 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:09.438689    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:56:10.446166 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:10.446130    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerStarted","Data":"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"}
Nov 03 23:56:11.333364 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:11.333258    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:56:11.334687 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:11.334656    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 23:56:11.334919 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:11.334887    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 23:56:12.332449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:12.331900    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 23:56:12.332449 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:12.331991    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 23:56:13.036617 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.036574    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 03 23:56:13.069971 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.063362    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 03 23:56:13.069971 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.063530    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.080685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.080655    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-2b1f66-serving-cert\""
Nov 03 23:56:13.131621 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.131568    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gn8l\" (UniqueName: \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.141607 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.141566    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.141937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.141910    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.142221 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.142197    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.269708 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.269670    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gn8l\" (UniqueName: \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.269916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.269776    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.269916 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.269851    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.270039 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.269970    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.270920 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.270895    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.274008 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.273983    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.275317 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.275290    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.293197 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.293139    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gn8l\" (UniqueName: \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\") pod \"maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.406100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.406069    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 23:56:13.420415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.412136    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 23:56:13.495478 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.495446    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:13.847848 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:13.847815    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 03 23:56:13.847973 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:56:13.847856    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5118d76b_260a_4cc7_9965_90df51e80bc9.slice/crio-91a27ab18b9474fe0d97ae3fd2bc295d136ed58bb9295b4a7e53073475f94ec1 WatchSource:0}: Error finding container 91a27ab18b9474fe0d97ae3fd2bc295d136ed58bb9295b4a7e53073475f94ec1: Status 404 returned error can't find the container with id 91a27ab18b9474fe0d97ae3fd2bc295d136ed58bb9295b4a7e53073475f94ec1
Nov 03 23:56:14.364070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:14.354586    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 23:56:14.364070 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:14.354649    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 23:56:14.448576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:14.448537    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:56:14.565206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:14.565180    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" event={"ID":"5118d76b-260a-4cc7-9965-90df51e80bc9","Type":"ContainerStarted","Data":"91a27ab18b9474fe0d97ae3fd2bc295d136ed58bb9295b4a7e53073475f94ec1"}
Nov 03 23:56:15.335411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:15.330620    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 23:56:15.335411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:15.330680    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 23:56:15.633466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:15.627618    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" event={"ID":"5118d76b-260a-4cc7-9965-90df51e80bc9","Type":"ContainerStarted","Data":"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"}
Nov 03 23:56:15.633466 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:15.629190    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:15.666209 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:15.666143    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" podStartSLOduration=3.130040627 podStartE2EDuration="3.666121878s" totalImagesPullingTime="536.081251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:15.662759342 +0000 UTC m=+1591.553821278" watchObservedRunningTime="2026-11-03 23:56:15.666121878 +0000 UTC m=+1591.557183814"
Nov 03 23:56:16.389239 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:16.389202    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:56:16.390827 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:16.390801    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 03 23:56:22.647191 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:22.647161    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 03 23:56:23.102613 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:23.102575    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:56:34.833238 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:56:34.833191    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:56:34.883111 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.883063    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 03 23:56:34.888002 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.887969    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.923059 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.923027    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 03 23:56:34.930056 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.930030    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a5948\"/\"auth-test-model-9a5948-kserve-self-signed-certs\""
Nov 03 23:56:34.930237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.930186    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a5948\"/\"openshift-service-ca.crt\""
Nov 03 23:56:34.930237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.930186    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a5948\"/\"default-dockercfg-z52dp\""
Nov 03 23:56:34.930237 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.930187    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a5948\"/\"kube-root-ca.crt\""
Nov 03 23:56:34.980257 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980219    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.980471 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980288    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.980605 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980584    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.980732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980708    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.980774 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980755    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kh6\" (UniqueName: \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:34.980822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:34.980790    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082439 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082364    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082732 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082716    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082818 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082797    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082880 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082803    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082880    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kh6\" (UniqueName: \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.082945 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.082933    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.083055 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.083026    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.083447 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.083347    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.085029 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.085005    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.085189 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.085167    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.096093 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.096065    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kh6\" (UniqueName: \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\") pod \"auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") " pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.214488 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.214161    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:35.696996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.696962    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 03 23:56:35.697460 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:56:35.697428    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c21546_658b_4453_b8d6_462984e9cdf0.slice/crio-abbfad33d708440bdaab6113ad750df498c809ff9704aaa49c699ac5bea0b2ef WatchSource:0}: Error finding container abbfad33d708440bdaab6113ad750df498c809ff9704aaa49c699ac5bea0b2ef: Status 404 returned error can't find the container with id abbfad33d708440bdaab6113ad750df498c809ff9704aaa49c699ac5bea0b2ef
Nov 03 23:56:35.880765 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.880731    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" event={"ID":"a0c21546-658b-4453-b8d6-462984e9cdf0","Type":"ContainerStarted","Data":"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"}
Nov 03 23:56:35.880765 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.880772    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" event={"ID":"a0c21546-658b-4453-b8d6-462984e9cdf0","Type":"ContainerStarted","Data":"abbfad33d708440bdaab6113ad750df498c809ff9704aaa49c699ac5bea0b2ef"}
Nov 03 23:56:35.906236 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:35.906187    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" podStartSLOduration=1.9061733840000001 podStartE2EDuration="1.906173384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:35.902361509 +0000 UTC m=+1611.793423444" watchObservedRunningTime="2026-11-03 23:56:35.906173384 +0000 UTC m=+1611.797235317"
Nov 03 23:56:36.215778 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:36.215743    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:41.503441 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:41.502474    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" podUID="b6a44b6090d9fc279a15fb8a9a158ae7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 03 23:56:43.994657 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:43.994620    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 03 23:56:43.998908 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:43.998882    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.004672 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.004633    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"kube-root-ca.crt\""
Nov 03 23:56:44.009495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.009467    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 03 23:56:44.009603 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.009497    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"default-dockercfg-nhbd9\""
Nov 03 23:56:44.009662 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.009645    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"auth-test-model-2341ca-kserve-self-signed-certs\""
Nov 03 23:56:44.104107 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.104077    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"openshift-service-ca.crt\""
Nov 03 23:56:44.138421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138360    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.138578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138480    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.138578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138509    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.138710 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138688    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.138809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138795    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zp2\" (UniqueName: \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.138844 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.138818    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.271249    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.271327    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.271358    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.272044    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.272274    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.272423    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zp2\" (UniqueName: \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.274460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.272453    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.276822 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.276795    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.287408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.281174    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.289729 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.289296    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"auth-test-model-2341ca-kserve-self-signed-certs\""
Nov 03 23:56:44.299520 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.298961    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.343296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.330570    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.343296 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.341995    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zp2\" (UniqueName: \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\") pod \"auth-test-model-2341ca-kserve-5866cbc745-z4crf\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") " pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.617568 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.617537    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2341ca\"/\"default-dockercfg-nhbd9\""
Nov 03 23:56:44.623625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.623602    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:44.985435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:44.984910    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 03 23:56:45.043503 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:45.043473    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" event={"ID":"369c472c-c589-48af-85c0-89320ae8ff57","Type":"ContainerStarted","Data":"70ec204cd85241b1c4a0e36ddac9f80dfed00a8c05c7e5f960439afdceaa48cf"}
Nov 03 23:56:45.216749 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:45.216720    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:45.229247 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:45.229223    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:46.052493 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:46.052445    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" event={"ID":"369c472c-c589-48af-85c0-89320ae8ff57","Type":"ContainerStarted","Data":"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"}
Nov 03 23:56:46.063750 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:46.063722    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 03 23:56:46.073348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:46.073303    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" podStartSLOduration=3.073292096 podStartE2EDuration="3.073292096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:46.072689086 +0000 UTC m=+1621.963751033" watchObservedRunningTime="2026-11-03 23:56:46.073292096 +0000 UTC m=+1621.964354029"
Nov 03 23:56:46.628410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:46.624499    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:54.624809 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:54.624724    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:54.637538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:54.637511    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:56:55.151698 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:56:55.151669    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 03 23:57:02.281205 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.281155    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 03 23:57:02.285348 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.285328    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.299784 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.299756    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9a5948-openshift-default-dockercfg-28jns\""
Nov 03 23:57:02.302740 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.302715    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.391930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.391992    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsp5\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392165    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392435    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392515    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392571    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392678    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392716    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.399435 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.392743    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494491    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494530 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494537    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494598    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494622    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsp5\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494768 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494755    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.494992 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494972    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.494971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495049 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495037    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495163 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495076    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495234    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495377 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495345    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495624 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495437    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.495748 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.495444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.497021 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.496999    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.497336 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.497316    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.507696 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.507667    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.507996 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.507974    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsp5\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.601169 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.601135    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:02.947023 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.946982    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 03 23:57:02.950884 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:57:02.950844    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e01cb9_1939_403b_9f99_467102f9417a.slice/crio-4a1a59f4725970d357b258bddc22e9835d6744c12e02a8c7e472257085480e01 WatchSource:0}: Error finding container 4a1a59f4725970d357b258bddc22e9835d6744c12e02a8c7e472257085480e01: Status 404 returned error can't find the container with id 4a1a59f4725970d357b258bddc22e9835d6744c12e02a8c7e472257085480e01
Nov 03 23:57:02.967246 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.967182    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:02.967585 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.967561    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:02.967764 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:02.967746    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:03.202411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:03.202306    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" event={"ID":"e2e01cb9-1939-403b-9f99-467102f9417a","Type":"ContainerStarted","Data":"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"}
Nov 03 23:57:03.202411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:03.202363    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" event={"ID":"e2e01cb9-1939-403b-9f99-467102f9417a","Type":"ContainerStarted","Data":"4a1a59f4725970d357b258bddc22e9835d6744c12e02a8c7e472257085480e01"}
Nov 03 23:57:03.230136 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:03.230074    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podStartSLOduration=1.230061601 podStartE2EDuration="1.230061601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:03.227032509 +0000 UTC m=+1639.118094465" watchObservedRunningTime="2026-11-03 23:57:03.230061601 +0000 UTC m=+1639.121123535"
Nov 03 23:57:03.612416 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:03.607519    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:04.602909 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:04.602869    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:57:04.603115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:04.602958    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 23:57:04.858227 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:57:04.858145    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:57:05.253685 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.253150    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 03 23:57:05.260330 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.260299    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.265965 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.265933    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 03 23:57:05.272933 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.272907    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2b1f66\"/\"kube-root-ca.crt\""
Nov 03 23:57:05.275034 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.275007    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2b1f66\"/\"default-dockercfg-vvxfb\""
Nov 03 23:57:05.275680 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.275658    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2b1f66\"/\"openshift-service-ca.crt\""
Nov 03 23:57:05.275801 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.275786    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2b1f66\"/\"auth-test-model-2b1f66-kserve-self-signed-certs\""
Nov 03 23:57:05.368618 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.368571    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.368842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.368685    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bjx\" (UniqueName: \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.368842 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.368771    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.368954 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.368908    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.369075 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.369057    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.369270 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.369243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.483530    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.483751    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.483833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bjx\" (UniqueName: \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.483908    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.483941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.484030    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.484914    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.485411 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.485169    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.497415 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.486698    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.505455 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.502162    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.547427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.531650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.547427 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.542086    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bjx\" (UniqueName: \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\") pod \"auth-test-model-2b1f66-kserve-b8795c965-p5qn9\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") " pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.593407 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.587583    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:05.660976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.659383    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:57:05.660976 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:05.660330    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 23:57:06.054087 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.054058    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 03 23:57:06.054460 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:57:06.054300    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6385f9dc_71ad_456b_a580_5693f3cf0770.slice/crio-21b0ea3aee403454eae9627b1a5953a165b955fef54c4391df8863ba14255189 WatchSource:0}: Error finding container 21b0ea3aee403454eae9627b1a5953a165b955fef54c4391df8863ba14255189: Status 404 returned error can't find the container with id 21b0ea3aee403454eae9627b1a5953a165b955fef54c4391df8863ba14255189
Nov 03 23:57:06.358462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.350122    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" event={"ID":"6385f9dc-71ad-456b-a580-5693f3cf0770","Type":"ContainerStarted","Data":"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"}
Nov 03 23:57:06.358462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.350165    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" event={"ID":"6385f9dc-71ad-456b-a580-5693f3cf0770","Type":"ContainerStarted","Data":"21b0ea3aee403454eae9627b1a5953a165b955fef54c4391df8863ba14255189"}
Nov 03 23:57:06.358462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.356350    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" podStartSLOduration=1.356331336 podStartE2EDuration="1.356331336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:06.351841652 +0000 UTC m=+1642.242903583" watchObservedRunningTime="2026-11-03 23:57:06.356331336 +0000 UTC m=+1642.247393274"
Nov 03 23:57:06.589538 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.589429    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:06.602871 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.602843    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:57:06.603028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:06.602893    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 23:57:07.601690 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:07.601655    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:57:07.602192 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:07.601716    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 23:57:08.601966 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:08.601932    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 23:57:08.602354 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:08.601987    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 23:57:09.606508 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:09.606474    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:09.606915 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:09.606887    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:09.607650 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:09.607618    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 03 23:57:15.589104 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:15.589062    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:15.601623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:15.601589    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:16.513462 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:16.513382    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 03 23:57:18.352496 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.352455    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 03 23:57:18.359159 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.359129    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.365131 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.365095    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 03 23:57:18.366421 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.366382    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"default-dockercfg-n7zls\""
Nov 03 23:57:18.369219 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.369195    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"auth-test-model-15b02a-kserve-self-signed-certs\""
Nov 03 23:57:18.371481 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.371456    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"openshift-service-ca.crt\""
Nov 03 23:57:18.371580 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.371553    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-15b02a\"/\"kube-root-ca.crt\""
Nov 03 23:57:18.501793 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.501757    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.501999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.501897    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkx5\" (UniqueName: \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.501999 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.501942    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.502110 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.502031    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.502170 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.502154    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.502215 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.502204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.611821 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.611676    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.619432 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.612126    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkx5\" (UniqueName: \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.619735 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.619704    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.620025 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.619999    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.625599 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.625569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.625830 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.625813    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.635507 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.626571    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.641550 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.629717    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.641550 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.629934    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.673410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.671123    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkx5\" (UniqueName: \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.673410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.671798    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.673410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.672000    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\") pod \"auth-test-model-15b02a-kserve-6c65764977-xt5k8\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") " pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:18.687044 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:18.686742    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:19.047718 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:19.047687    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 03 23:57:19.048536 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:57:19.048505    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod382849c4_a50a_43bd_80ea_1e1b147cb976.slice/crio-f1551f2b9bc5796f6f84c53bd62ea722d2a931e5feaacc3f06189855b4622cd2 WatchSource:0}: Error finding container f1551f2b9bc5796f6f84c53bd62ea722d2a931e5feaacc3f06189855b4622cd2: Status 404 returned error can't find the container with id f1551f2b9bc5796f6f84c53bd62ea722d2a931e5feaacc3f06189855b4622cd2
Nov 03 23:57:19.473973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:19.473938    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" event={"ID":"382849c4-a50a-43bd-80ea-1e1b147cb976","Type":"ContainerStarted","Data":"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"}
Nov 03 23:57:19.473973 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:19.473980    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" event={"ID":"382849c4-a50a-43bd-80ea-1e1b147cb976","Type":"ContainerStarted","Data":"f1551f2b9bc5796f6f84c53bd62ea722d2a931e5feaacc3f06189855b4622cd2"}
Nov 03 23:57:19.498820 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:19.498756    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" podStartSLOduration=1.498741455 podStartE2EDuration="1.498741455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:19.496012636 +0000 UTC m=+1655.387074574" watchObservedRunningTime="2026-11-03 23:57:19.498741455 +0000 UTC m=+1655.389803388"
Nov 03 23:57:19.696408 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:19.689487    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:20.508022 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:20.507988    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:57:28.721410 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:28.714077    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:28.794064 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:28.794036    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:28.805947 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:28.805921    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 03 23:57:34.833801 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:57:34.833762    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:57:37.820005 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.819968    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 03 23:57:37.824625 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.824594    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.832635 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.832586    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2b1f66-openshift-default-dockercfg-wrq2t\""
Nov 03 23:57:37.837206 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.837182    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 03 23:57:37.933063 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933031    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933090    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933271 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933177    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933476 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933324    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933521 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933484    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdnr\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933564 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933539    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933601    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933791 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933775    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:37.933860 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:37.933827    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.034926 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.034892    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.034941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035100 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035054    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035218 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035147    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035273 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035237    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdnr\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035273 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035266    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035380 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035298    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035518 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035578 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035558    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035672    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035693 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035680    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035679    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035813 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035780    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.035937 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.035920    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.037724 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.037694    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.037882 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.037861    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.048006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.047984    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.048254 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.048232    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdnr\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.142741 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.142655    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:38.544495 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.544468    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 03 23:57:38.545521 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:57:38.545490    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b36b6d_b234_43ce_894d_de820ad04451.slice/crio-24e2b65f695f25d149fb6be6f9ccd09927dd640ba1ceee32e2f887319386a7dc WatchSource:0}: Error finding container 24e2b65f695f25d149fb6be6f9ccd09927dd640ba1ceee32e2f887319386a7dc: Status 404 returned error can't find the container with id 24e2b65f695f25d149fb6be6f9ccd09927dd640ba1ceee32e2f887319386a7dc
Nov 03 23:57:38.548195 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.548163    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:38.548277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.548231    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:38.548277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.548263    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:57:38.863745 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.863705    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" event={"ID":"67b36b6d-b234-43ce-894d-de820ad04451","Type":"ContainerStarted","Data":"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"}
Nov 03 23:57:38.864140 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:38.863805    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" event={"ID":"67b36b6d-b234-43ce-894d-de820ad04451","Type":"ContainerStarted","Data":"24e2b65f695f25d149fb6be6f9ccd09927dd640ba1ceee32e2f887319386a7dc"}
Nov 03 23:57:39.143334 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:39.143240    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:39.145289 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:39.145258    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:39.145469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:39.145314    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:40.143047 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:40.143011    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:40.143590 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:40.143063    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:41.143329 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:41.143294    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:41.143782 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:41.143400    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:42.143859 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:42.143821    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:42.144231 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:42.143886    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:43.143227 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:43.143192    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:43.143487 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:43.143264    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:44.144565 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:44.144527    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 23:57:44.145192 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:44.145160    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 23:57:45.148888 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:45.148856    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:45.149370 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:45.149315    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:45.150154 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:45.150134    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 03 23:57:45.179277 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:57:45.179226    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podStartSLOduration=8.179213555 podStartE2EDuration="8.179213555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:38.891689208 +0000 UTC m=+1674.782751143" watchObservedRunningTime="2026-11-03 23:57:45.179213555 +0000 UTC m=+1681.070275489"
Nov 03 23:58:04.834567 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:58:04.834528    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:58:10.402988 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:58:10.402950    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:58:34.833837 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:58:34.833804    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:58:57.502944 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:58:57.502912    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 03 23:59:04.833252 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:59:04.833216    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:59:22.830359 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.830326    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 03 23:59:22.835598 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.835576    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.843127 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.843101    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2341ca-openshift-default-dockercfg-s4vcn\""
Nov 03 23:59:22.848970 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.848944    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 03 23:59:22.907836 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.907802    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpcq\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908001 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.907931    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908008    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908053 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908031    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908255 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908231    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908326 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908269    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908460 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908437    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908537 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908520    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:22.908576 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:22.908566    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.009998    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010032 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010037    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010082    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010132    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010168    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010279 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpcq\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010340    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010513 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010522    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010542    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010623 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010578    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.010930 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010904    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.011031 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.010958    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.011234 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.011208    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.012543 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.012521    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.012849 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.012825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.022964 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.022935    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.023115 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.023099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpcq\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.152403 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.152312    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:23.443446 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.443371    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 03 23:59:23.444379 ip-10-0-143-134 kubenswrapper[6430]: W1103 23:59:23.444348    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a1d1ca_ecec_41a0_9642_e41e84f9dd1d.slice/crio-a38a96041cb72de832a87b97a9863bac8cf43c415cfaff05f80e8c5c3da40d37 WatchSource:0}: Error finding container a38a96041cb72de832a87b97a9863bac8cf43c415cfaff05f80e8c5c3da40d37: Status 404 returned error can't find the container with id a38a96041cb72de832a87b97a9863bac8cf43c415cfaff05f80e8c5c3da40d37
Nov 03 23:59:23.447469 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.447419    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:59:23.447544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.447502    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:59:23.447544 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:23.447529    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 23:59:24.475479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:24.473677    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" event={"ID":"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d","Type":"ContainerStarted","Data":"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"}
Nov 03 23:59:24.475479 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:24.473755    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" event={"ID":"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d","Type":"ContainerStarted","Data":"a38a96041cb72de832a87b97a9863bac8cf43c415cfaff05f80e8c5c3da40d37"}
Nov 03 23:59:25.159429 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:25.154598    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:25.174412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:25.171562    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:25.174412 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:25.171651    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:26.152924 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:26.152887    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:26.153320 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:26.152945    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:27.152853 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:27.152823    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:27.153028 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:27.152883    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:28.152582 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:28.152552    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:28.152775 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:28.152606    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:29.155313 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:29.155275    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:29.156006 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:29.155343    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:30.152736 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:30.152699    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 23:59:30.152942 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:30.152768    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 23:59:31.157504 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:31.157474    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:31.158048 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:31.158026    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:31.158657 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:31.158634    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 03 23:59:31.192016 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:31.191959    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podStartSLOduration=9.191940714 podStartE2EDuration="9.191940714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:59:24.521618512 +0000 UTC m=+1780.412680457" watchObservedRunningTime="2026-11-03 23:59:31.191940714 +0000 UTC m=+1787.083002650"
Nov 03 23:59:34.832191 ip-10-0-143-134 kubenswrapper[6430]: E1103 23:59:34.832147    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 03 23:59:47.262184 ip-10-0-143-134 kubenswrapper[6430]: I1103 23:59:47.262149    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:00:01.593683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.593593    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"]
Nov 04 00:00:01.599896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.599869    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.605977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.605941    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"]
Nov 04 00:00:01.607873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.607847    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:01.607990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.607940    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:00:01.667460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.667399    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.667638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.667600    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbjk\" (UniqueName: \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.667794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.667775    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.769370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.769310    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbjk\" (UniqueName: \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.769690 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.769497    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.769867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.769761    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.770827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.770803    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.772241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.772215    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.782536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.782511    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbjk\" (UniqueName: \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\") pod \"maas-api-revoke-sub-783d53cd-tqvpg\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:01.957459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:01.956132    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:02.263843 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:00:02.263803    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c8fe0_5cd7_418d_8104_286f072e8f34.slice/crio-0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8 WatchSource:0}: Error finding container 0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8: Status 404 returned error can't find the container with id 0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8
Nov 04 00:00:02.264938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:02.264911    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"]
Nov 04 00:00:02.962044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:02.962001    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg" event={"ID":"3b1c8fe0-5cd7-418d-8104-286f072e8f34","Type":"ContainerStarted","Data":"8366eefc9a599fff14205559af8f70990d15c66554d3ab9a8be4d58149f4f7c1"}
Nov 04 00:00:02.962477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:02.962065    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg" event={"ID":"3b1c8fe0-5cd7-418d-8104-286f072e8f34","Type":"ContainerStarted","Data":"0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8"}
Nov 04 00:00:02.985098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:02.985046    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg" podStartSLOduration=1.985031373 podStartE2EDuration="1.985031373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.983661597 +0000 UTC m=+1818.874723534" watchObservedRunningTime="2026-11-04 00:00:02.985031373 +0000 UTC m=+1818.876093306"
Nov 04 00:00:03.972892 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:03.972854    6430 generic.go:356] "Generic (PLEG): container finished" podID="3b1c8fe0-5cd7-418d-8104-286f072e8f34" containerID="8366eefc9a599fff14205559af8f70990d15c66554d3ab9a8be4d58149f4f7c1" exitCode=0
Nov 04 00:00:03.973305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:03.972937    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg" event={"ID":"3b1c8fe0-5cd7-418d-8104-286f072e8f34","Type":"ContainerDied","Data":"8366eefc9a599fff14205559af8f70990d15c66554d3ab9a8be4d58149f4f7c1"}
Nov 04 00:00:04.845019 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:04.844978    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:00:05.153301 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.153272    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:05.268325 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.268291    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\") pod \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") "
Nov 04 00:00:05.268530 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.268348    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\" (UniqueName: \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\") pod \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") "
Nov 04 00:00:05.268530 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.268492    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\") pod \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\" (UID: \"3b1c8fe0-5cd7-418d-8104-286f072e8f34\") "
Nov 04 00:00:05.268710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.268689    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca" pod "3b1c8fe0-5cd7-418d-8104-286f072e8f34" (UID: "3b1c8fe0-5cd7-418d-8104-286f072e8f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:05.269141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.269116    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:05.270708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.270684    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk" pod "3b1c8fe0-5cd7-418d-8104-286f072e8f34" (UID: "3b1c8fe0-5cd7-418d-8104-286f072e8f34"). InnerVolumeSpecName "kube-api-access-qvbjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:05.270808 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.270724    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca" pod "3b1c8fe0-5cd7-418d-8104-286f072e8f34" (UID: "3b1c8fe0-5cd7-418d-8104-286f072e8f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:05.371224 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.371143    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1c8fe0-5cd7-418d-8104-286f072e8f34-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:05.371224 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.371174    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbjk\" (UniqueName: \"kubernetes.io/projected/3b1c8fe0-5cd7-418d-8104-286f072e8f34-kube-api-access-qvbjk\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:05.990377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.990349    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"
Nov 04 00:00:05.990377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.990366    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg" event={"ID":"3b1c8fe0-5cd7-418d-8104-286f072e8f34","Type":"ContainerDied","Data":"0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8"}
Nov 04 00:00:05.990619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:05.990445    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db7f8db1ed102d60060ca7a65a9767b46ba1c607236b6e972d4c467881517d8"
Nov 04 00:00:11.941453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.941413    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"]
Nov 04 00:00:11.942257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.942241    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3b1c8fe0-5cd7-418d-8104-286f072e8f34" containerName="revoke-keys"
Nov 04 00:00:11.942301 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.942258    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1c8fe0-5cd7-418d-8104-286f072e8f34" containerName="revoke-keys"
Nov 04 00:00:11.942432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.942421    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1c8fe0-5cd7-418d-8104-286f072e8f34" containerName="revoke-keys"
Nov 04 00:00:11.946283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.946266    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:11.954020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.953989    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"]
Nov 04 00:00:11.955196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.955173    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:00:11.955382 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.955187    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:00:11.959219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.959190    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:11.959330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.959267    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:11.959553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:11.959528    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z77f\" (UniqueName: \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.097427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.084425    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z77f\" (UniqueName: \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.097427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.084689    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.097427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.084777    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.097427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.087684    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.130044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.130009    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z77f\" (UniqueName: \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.131472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.131444    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d90d111-cbgfg\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.266348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.266264    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:12.553553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:12.553526    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"]
Nov 04 00:00:12.553942 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:00:12.553914    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ea8423_e44d_438e_9c5d_ede8931cfffa.slice/crio-5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b WatchSource:0}: Error finding container 5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b: Status 404 returned error can't find the container with id 5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b
Nov 04 00:00:13.169980 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:13.169939    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg" event={"ID":"59ea8423-e44d-438e-9c5d-ede8931cfffa","Type":"ContainerStarted","Data":"1a83b45d21f877ac27b82a22dde6a6c7f2cee376cbdd1a5301314de73537fcbf"}
Nov 04 00:00:13.170354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:13.169997    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg" event={"ID":"59ea8423-e44d-438e-9c5d-ede8931cfffa","Type":"ContainerStarted","Data":"5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b"}
Nov 04 00:00:13.193040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:13.192992    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg" podStartSLOduration=2.192975432 podStartE2EDuration="2.192975432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:13.190811507 +0000 UTC m=+1829.081873438" watchObservedRunningTime="2026-11-04 00:00:13.192975432 +0000 UTC m=+1829.084037366"
Nov 04 00:00:14.179928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:14.179892    6430 generic.go:356] "Generic (PLEG): container finished" podID="59ea8423-e44d-438e-9c5d-ede8931cfffa" containerID="1a83b45d21f877ac27b82a22dde6a6c7f2cee376cbdd1a5301314de73537fcbf" exitCode=0
Nov 04 00:00:14.180426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:14.180000    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg" event={"ID":"59ea8423-e44d-438e-9c5d-ede8931cfffa","Type":"ContainerDied","Data":"1a83b45d21f877ac27b82a22dde6a6c7f2cee376cbdd1a5301314de73537fcbf"}
Nov 04 00:00:15.359810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.359784    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:15.456960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.456869    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\") pod \"59ea8423-e44d-438e-9c5d-ede8931cfffa\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") "
Nov 04 00:00:15.456960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.456934    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\" (UniqueName: \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\") pod \"59ea8423-e44d-438e-9c5d-ede8931cfffa\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") "
Nov 04 00:00:15.457202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.456986    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\") pod \"59ea8423-e44d-438e-9c5d-ede8931cfffa\" (UID: \"59ea8423-e44d-438e-9c5d-ede8931cfffa\") "
Nov 04 00:00:15.457286 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.457261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca" pod "59ea8423-e44d-438e-9c5d-ede8931cfffa" (UID: "59ea8423-e44d-438e-9c5d-ede8931cfffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:15.457810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.457776    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:15.459241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.459213    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f" pod "59ea8423-e44d-438e-9c5d-ede8931cfffa" (UID: "59ea8423-e44d-438e-9c5d-ede8931cfffa"). InnerVolumeSpecName "kube-api-access-7z77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:15.459463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.459379    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca" pod "59ea8423-e44d-438e-9c5d-ede8931cfffa" (UID: "59ea8423-e44d-438e-9c5d-ede8931cfffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:15.559327 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.559290    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z77f\" (UniqueName: \"kubernetes.io/projected/59ea8423-e44d-438e-9c5d-ede8931cfffa-kube-api-access-7z77f\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:15.559579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:15.559334    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ea8423-e44d-438e-9c5d-ede8931cfffa-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:16.195623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.195594    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"
Nov 04 00:00:16.195623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.195606    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg" event={"ID":"59ea8423-e44d-438e-9c5d-ede8931cfffa","Type":"ContainerDied","Data":"5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b"}
Nov 04 00:00:16.195864 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.195689    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc9e75e6255426f4b41233031e379787f3517c985c8dbf7e620ba867aa56e2b"
Nov 04 00:00:16.266155 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.266122    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 04 00:00:16.266335 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.266312    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy" containerID="cri-o://8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa" gracePeriod=30
Nov 04 00:00:16.269889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.269866    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 04 00:00:16.270049 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:16.270027    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy" containerID="cri-o://dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53" gracePeriod=30
Nov 04 00:00:17.628419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:17.628368    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 04 00:00:17.630520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:17.630467    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy" containerID="cri-o://f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa" gracePeriod=30
Nov 04 00:00:17.633897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:17.633875    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 04 00:00:17.634134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:17.634099    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy" containerID="cri-o://ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f" gracePeriod=30
Nov 04 00:00:21.553611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.553583    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 04 00:00:21.570061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570032    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570193    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570234    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570267    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570367    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570437    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570520    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.570739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570574    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.570739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.570739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570662    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\") pod \"a655dc6a-85fc-4e2a-b956-57020d714d37\" (UID: \"a655dc6a-85fc-4e2a-b956-57020d714d37\") "
Nov 04 00:00:21.571001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570782    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.571001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570805    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.571001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.570814    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:21.571685 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.571657    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a655dc6a-85fc-4e2a-b956-57020d714d37-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.571792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.571687    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.571792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.571702    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.571792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.571715    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.571792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.571728    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.572253 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.572225    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:21.572645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.572627    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "kube-api-access-sl2f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:21.572729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.572712    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:21.572850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.572835    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy" pod "a655dc6a-85fc-4e2a-b956-57020d714d37" (UID: "a655dc6a-85fc-4e2a-b956-57020d714d37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.673695 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.673663    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.673695 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.673691    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.673899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.673705    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a655dc6a-85fc-4e2a-b956-57020d714d37-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.673899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.673720    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2f6\" (UniqueName: \"kubernetes.io/projected/a655dc6a-85fc-4e2a-b956-57020d714d37-kube-api-access-sl2f6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.954681 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.954656    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 04 00:00:21.978151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978124    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978164    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978202    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978266    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978316    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978480    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978541    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978556    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.978717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978673    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.978717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978699    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978737    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\") pod \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\" (UID: \"f17c35ea-4a14-480e-af95-72c14ae0d6dc\") "
Nov 04 00:00:21.978868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.978926 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.978875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:21.979140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979115    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.979731 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979705    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.979820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979735    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.979820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979750    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.979820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979764    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.979820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.979778    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:21.980538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.980511    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:21.980636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.980601    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:21.980674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.980650    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:21.980777 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:21.980762    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4" pod "f17c35ea-4a14-480e-af95-72c14ae0d6dc" (UID: "f17c35ea-4a14-480e-af95-72c14ae0d6dc"). InnerVolumeSpecName "kube-api-access-gkbt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:22.081318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.081225    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbt4\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-kube-api-access-gkbt4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.081318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.081267    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.081318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.081282    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.081318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.081296    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17c35ea-4a14-480e-af95-72c14ae0d6dc-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:22.319459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.319362    6430 generic.go:356] "Generic (PLEG): container finished" podID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerID="dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53" exitCode=0
Nov 04 00:00:22.319582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.319477    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"
Nov 04 00:00:22.319838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.319439    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" event={"ID":"a655dc6a-85fc-4e2a-b956-57020d714d37","Type":"ContainerDied","Data":"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"}
Nov 04 00:00:22.319838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.319708    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg" event={"ID":"a655dc6a-85fc-4e2a-b956-57020d714d37","Type":"ContainerDied","Data":"d59c8402a385d01c0dd4d0e23bd85d4a50f58d1fef8ebf655dee48674c882ee7"}
Nov 04 00:00:22.319838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.319732    6430 scope.go:122] "RemoveContainer" containerID="dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"
Nov 04 00:00:22.321427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.321365    6430 generic.go:356] "Generic (PLEG): container finished" podID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerID="8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa" exitCode=0
Nov 04 00:00:22.321516 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.321456    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"
Nov 04 00:00:22.321679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.321596    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" event={"ID":"f17c35ea-4a14-480e-af95-72c14ae0d6dc","Type":"ContainerDied","Data":"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"}
Nov 04 00:00:22.321679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.321649    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7" event={"ID":"f17c35ea-4a14-480e-af95-72c14ae0d6dc","Type":"ContainerDied","Data":"c8ae5bc41ee5a3de9dadd828acc8909f9d63659af9f3a4ed31fa3f09455d19c0"}
Nov 04 00:00:22.332078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.332018    6430 scope.go:122] "RemoveContainer" containerID="dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"
Nov 04 00:00:22.332354 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:22.332326    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53\": container with ID starting with dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53 not found: ID does not exist" containerID="dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"
Nov 04 00:00:22.332474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.332363    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53"} err="failed to get container status \"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53\": rpc error: code = NotFound desc = could not find container \"dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53\": container with ID starting with dd582017a9920fe61d05501d4705183abedb5fae44d4b4a5ba4713f203baab53 not found: ID does not exist"
Nov 04 00:00:22.332474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.332400    6430 scope.go:122] "RemoveContainer" containerID="8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.426516    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.486707    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-vxrfg"]
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.486753    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.486775    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-dnzz7"]
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.428758    6430 scope.go:122] "RemoveContainer" containerID="8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:22.487326    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa\": container with ID starting with 8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa not found: ID does not exist" containerID="8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"
Nov 04 00:00:22.487582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.487376    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa"} err="failed to get container status \"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa\": rpc error: code = NotFound desc = could not find container \"8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa\": container with ID starting with 8813d72647a652a8a66175f04751851d8fbc2e0d4b8a2236183ca934062dabaa not found: ID does not exist"
Nov 04 00:00:22.961931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.961878    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 04 00:00:22.970852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:22.970052    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 04 00:00:23.039287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039256    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.039518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039305    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.039518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039334    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.039518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039363    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.039518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039445    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039523    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039617    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039656    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.039751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039739    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039806    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039852    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039867    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039920    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039924    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.039980    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.040058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040016    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.040592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040175    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.040592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040078    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040280    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040315    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\") pod \"1fb12ba6-dce0-4c74-b25e-073043cbc811\" (UID: \"1fb12ba6-dce0-4c74-b25e-073043cbc811\") "
Nov 04 00:00:23.040592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040400    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\") pod \"5e409ddd-715f-4bb2-8151-ab26f2705c88\" (UID: \"5e409ddd-715f-4bb2-8151-ab26f2705c88\") "
Nov 04 00:00:23.040953 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040730    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.040953 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.040825    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041428    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041456    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041472    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041485    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041500    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041515    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e409ddd-715f-4bb2-8151-ab26f2705c88-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041529    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.041955    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.042117    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.042123    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.042123    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:23.042218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.042193    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.043489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043457    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "kube-api-access-phmbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:23.043489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043480    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.043644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043499    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "kube-api-access-pr6q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:23.043644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043482    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:23.043644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043535    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo" pod "5e409ddd-715f-4bb2-8151-ab26f2705c88" (UID: "5e409ddd-715f-4bb2-8151-ab26f2705c88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:23.043766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.043740    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy" pod "1fb12ba6-dce0-4c74-b25e-073043cbc811" (UID: "1fb12ba6-dce0-4c74-b25e-073043cbc811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:23.143598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143566    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143595    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143608    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmbd\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-kube-api-access-phmbd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143621    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr6q7\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-kube-api-access-pr6q7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143630    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb12ba6-dce0-4c74-b25e-073043cbc811-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143640    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb12ba6-dce0-4c74-b25e-073043cbc811-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143652    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143660    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143669    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb12ba6-dce0-4c74-b25e-073043cbc811-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.143897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.143676    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e409ddd-715f-4bb2-8151-ab26f2705c88-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:23.332035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.332001    6430 generic.go:356] "Generic (PLEG): container finished" podID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerID="f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa" exitCode=0
Nov 04 00:00:23.332191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.332055    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"
Nov 04 00:00:23.332191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.332084    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" event={"ID":"1fb12ba6-dce0-4c74-b25e-073043cbc811","Type":"ContainerDied","Data":"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"}
Nov 04 00:00:23.332191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.332135    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x" event={"ID":"1fb12ba6-dce0-4c74-b25e-073043cbc811","Type":"ContainerDied","Data":"d2754aa0644dcbfb11ff9173fc1062d24dd2b47c19f7d3e8900b8fb9dddd9c02"}
Nov 04 00:00:23.332191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.332162    6430 scope.go:122] "RemoveContainer" containerID="f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"
Nov 04 00:00:23.333768 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.333742    6430 generic.go:356] "Generic (PLEG): container finished" podID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerID="ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f" exitCode=0
Nov 04 00:00:23.333876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.333795    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"
Nov 04 00:00:23.333876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.333832    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" event={"ID":"5e409ddd-715f-4bb2-8151-ab26f2705c88","Type":"ContainerDied","Data":"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"}
Nov 04 00:00:23.333980 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.333879    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8" event={"ID":"5e409ddd-715f-4bb2-8151-ab26f2705c88","Type":"ContainerDied","Data":"5f71463d9c0efa71f9d1f284c53fab932d68d05d53f4e3665f080ec79da0c094"}
Nov 04 00:00:23.343649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.343560    6430 scope.go:122] "RemoveContainer" containerID="f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"
Nov 04 00:00:23.343891 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:23.343867    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa\": container with ID starting with f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa not found: ID does not exist" containerID="f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"
Nov 04 00:00:23.343975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.343897    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa"} err="failed to get container status \"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa\": rpc error: code = NotFound desc = could not find container \"f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa\": container with ID starting with f0b42e167be41c4c0d844b89091aa3c894fc4ee1a518ddff3b55716f5ae293fa not found: ID does not exist"
Nov 04 00:00:23.343975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.343919    6430 scope.go:122] "RemoveContainer" containerID="ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"
Nov 04 00:00:23.353329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.353314    6430 scope.go:122] "RemoveContainer" containerID="ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"
Nov 04 00:00:23.353681 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:23.353665    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f\": container with ID starting with ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f not found: ID does not exist" containerID="ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"
Nov 04 00:00:23.353753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.353686    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f"} err="failed to get container status \"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f\": rpc error: code = NotFound desc = could not find container \"ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f\": container with ID starting with ee5bd188ca20822f9b56960bd248ed93d67370591450c2a9b4d1daa0ebdd788f not found: ID does not exist"
Nov 04 00:00:23.373410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.373375    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 04 00:00:23.376870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.376851    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-sfzg8"]
Nov 04 00:00:23.400534 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.400514    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 04 00:00:23.404360 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:23.404337    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-kx66x"]
Nov 04 00:00:24.244993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:24.244961    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" path="/var/lib/kubelet/pods/1fb12ba6-dce0-4c74-b25e-073043cbc811/volumes"
Nov 04 00:00:24.245510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:24.245495    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" path="/var/lib/kubelet/pods/5e409ddd-715f-4bb2-8151-ab26f2705c88/volumes"
Nov 04 00:00:24.245938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:24.245923    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" path="/var/lib/kubelet/pods/a655dc6a-85fc-4e2a-b956-57020d714d37/volumes"
Nov 04 00:00:24.246698 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:24.246682    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" path="/var/lib/kubelet/pods/f17c35ea-4a14-480e-af95-72c14ae0d6dc/volumes"
Nov 04 00:00:26.209802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:26.209767    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.389433    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391232    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391253    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391297    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391305    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391325    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391332    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391432    6430 cpu_manager.go:414] "containerMap: removing container" podUID="59ea8423-e44d-438e-9c5d-ede8931cfffa" containerName="revoke-keys"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391441    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ea8423-e44d-438e-9c5d-ede8931cfffa" containerName="revoke-keys"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391535    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391542    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391869    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17c35ea-4a14-480e-af95-72c14ae0d6dc" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391887    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb12ba6-dce0-4c74-b25e-073043cbc811" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391908    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e409ddd-715f-4bb2-8151-ab26f2705c88" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391917    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a655dc6a-85fc-4e2a-b956-57020d714d37" containerName="istio-proxy"
Nov 04 00:00:27.394654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.391948    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ea8423-e44d-438e-9c5d-ede8931cfffa" containerName="revoke-keys"
Nov 04 00:00:27.403265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.403237    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.410653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.410626    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:27.424957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.424932    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ba68e7-openshift-default-dockercfg-s6c4g\""
Nov 04 00:00:27.484958 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.484907    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485068    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485130    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l6g\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485180    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485296    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485340    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485482    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485513    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.485636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.485539    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587466    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587552    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l6g\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587602    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587714    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587756    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587861    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587891    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587916    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.587991    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.588648    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.603723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.588884    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.604876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.604843    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.606853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.605981    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.607008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.606646    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.676512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.664933    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.676512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.665484    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.688431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.678139    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.707442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.698101    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l6g\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\") pod \"e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") " pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:27.742423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:27.740443    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:28.161534 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.161505    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:28.162070 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:00:28.162049    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cd1d43_e19c_4b3a_ac86_8b6177cfe1fb.slice/crio-9d1d1f25c29751491effab27cbd54fab67d0357cf6cba7a61b38c65866b222c3 WatchSource:0}: Error finding container 9d1d1f25c29751491effab27cbd54fab67d0357cf6cba7a61b38c65866b222c3: Status 404 returned error can't find the container with id 9d1d1f25c29751491effab27cbd54fab67d0357cf6cba7a61b38c65866b222c3
Nov 04 00:00:28.164754 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.164722    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:28.164817 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.164789    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:28.164858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.164816    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:28.668136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.668097    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" event={"ID":"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb","Type":"ContainerStarted","Data":"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"}
Nov 04 00:00:28.668518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.668157    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" event={"ID":"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb","Type":"ContainerStarted","Data":"9d1d1f25c29751491effab27cbd54fab67d0357cf6cba7a61b38c65866b222c3"}
Nov 04 00:00:28.698594 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.698540    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" podStartSLOduration=1.6985269299999999 podStartE2EDuration="1.69852693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:28.694564468 +0000 UTC m=+1844.585626407" watchObservedRunningTime="2026-11-04 00:00:28.69852693 +0000 UTC m=+1844.589588863"
Nov 04 00:00:28.741865 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:28.741843    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:29.747724 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:29.747687    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:30.683050 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:30.683009    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:30.684045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:30.684017    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:34.832147 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:34.832111    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:00:36.654740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:36.654703    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:00:38.845410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:38.836828    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": context deadline exceeded"
Nov 04 00:00:39.006146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.006108    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:00:39.023582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.023548    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.048616 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.048586    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-fa0d0a-serving-cert\""
Nov 04 00:00:39.049869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.042069    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:00:39.079991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.079951    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gd79\" (UniqueName: \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.080266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.080243    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.080530 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.080510    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.080850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.080827    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.183013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.182925    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.183181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.183065    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.183251 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.183223    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gd79\" (UniqueName: \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.183251 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.183242    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.183733 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.183701    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.185694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.185669    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.185833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.185716    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.197576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.197550    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gd79\" (UniqueName: \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\") pod \"maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.369125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.369071    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:39.663146 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:00:39.663104    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d51fc39_822d_4d36_af2f_5dd386f57a78.slice/crio-d1de392566b11c61fd487fe0be8c74296cba71cc9aea66fba1b3011834f6dd0f WatchSource:0}: Error finding container d1de392566b11c61fd487fe0be8c74296cba71cc9aea66fba1b3011834f6dd0f: Status 404 returned error can't find the container with id d1de392566b11c61fd487fe0be8c74296cba71cc9aea66fba1b3011834f6dd0f
Nov 04 00:00:39.663964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.663938    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:00:39.976414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:39.974662    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" event={"ID":"1d51fc39-822d-4d36-af2f-5dd386f57a78","Type":"ContainerStarted","Data":"d1de392566b11c61fd487fe0be8c74296cba71cc9aea66fba1b3011834f6dd0f"}
Nov 04 00:00:40.986561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:40.986474    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" event={"ID":"1d51fc39-822d-4d36-af2f-5dd386f57a78","Type":"ContainerStarted","Data":"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"}
Nov 04 00:00:40.986925 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:40.986631    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:41.010830 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.010764    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" podStartSLOduration=2.071171378 podStartE2EDuration="3.010748465s" totalImagesPullingTime="939.577087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:41.007960358 +0000 UTC m=+1856.899022286" watchObservedRunningTime="2026-11-04 00:00:41.010748465 +0000 UTC m=+1856.901810411"
Nov 04 00:00:41.233600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.233555    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerName="main" containerID="cri-o://a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c" gracePeriod=45
Nov 04 00:00:41.534175 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.534143    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 04 00:00:41.682827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.682789    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683010 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.682906    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683079 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683060    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:41.683135 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683120    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683177 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683151    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683246    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\" (UniqueName: \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683297    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\" (UniqueName: \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\") pod \"382849c4-a50a-43bd-80ea-1e1b147cb976\" (UID: \"382849c4-a50a-43bd-80ea-1e1b147cb976\") "
Nov 04 00:00:41.683537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683445    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:41.683578 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683535    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:41.683979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683956    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:41.684064 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683982    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:41.684064 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.683993    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:41.685216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.685185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:41.685427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.685399    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:00:41.685515 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.685457    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5" pod "382849c4-a50a-43bd-80ea-1e1b147cb976" (UID: "382849c4-a50a-43bd-80ea-1e1b147cb976"). InnerVolumeSpecName "kube-api-access-7zkx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:41.785266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.785171    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382849c4-a50a-43bd-80ea-1e1b147cb976-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:41.785266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.785210    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zkx5\" (UniqueName: \"kubernetes.io/projected/382849c4-a50a-43bd-80ea-1e1b147cb976-kube-api-access-7zkx5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:41.785266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:41.785222    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382849c4-a50a-43bd-80ea-1e1b147cb976-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:42.064109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.061285    6430 generic.go:356] "Generic (PLEG): container finished" podID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerID="a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c" exitCode=0
Nov 04 00:00:42.085411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.073604    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"
Nov 04 00:00:42.085411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.079013    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" event={"ID":"382849c4-a50a-43bd-80ea-1e1b147cb976","Type":"ContainerDied","Data":"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"}
Nov 04 00:00:42.085411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.079072    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8" event={"ID":"382849c4-a50a-43bd-80ea-1e1b147cb976","Type":"ContainerDied","Data":"f1551f2b9bc5796f6f84c53bd62ea722d2a931e5feaacc3f06189855b4622cd2"}
Nov 04 00:00:42.085411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.079099    6430 scope.go:122] "RemoveContainer" containerID="a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"
Nov 04 00:00:42.133694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.130362    6430 scope.go:122] "RemoveContainer" containerID="a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"
Nov 04 00:00:42.134138 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:42.134022    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c\": container with ID starting with a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c not found: ID does not exist" containerID="a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"
Nov 04 00:00:42.134138 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.134063    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c"} err="failed to get container status \"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c\": rpc error: code = NotFound desc = could not find container \"a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c\": container with ID starting with a1300607ffa7235b41b46f285c02a8a4d1e97b4703d3680ee92bbfa56d8d791c not found: ID does not exist"
Nov 04 00:00:42.178626 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.178591    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 04 00:00:42.222199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.221015    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-15b02a/auth-test-model-15b02a-kserve-6c65764977-xt5k8"]
Nov 04 00:00:42.259431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:42.254533    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" path="/var/lib/kubelet/pods/382849c4-a50a-43bd-80ea-1e1b147cb976/volumes"
Nov 04 00:00:46.487743 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:46.487698    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 04 00:00:47.079683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:47.079645    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:00:49.228566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.228530    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:49.229756 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.229703    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" podUID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" containerName="istio-proxy" containerID="cri-o://6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2" gracePeriod=30
Nov 04 00:00:49.572421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.572258    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 04 00:00:49.574524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.572754    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy" containerID="cri-o://840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75" gracePeriod=30
Nov 04 00:00:49.642063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.642005    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 04 00:00:49.642304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:49.642272    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy" containerID="cri-o://e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102" gracePeriod=30
Nov 04 00:00:50.399226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:50.399187    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 04 00:00:50.399757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:50.399474    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy" containerID="cri-o://124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2" gracePeriod=30
Nov 04 00:00:51.249572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.249531    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:00:51.250765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.250737    6430 cpu_manager.go:414] "containerMap: removing container" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerName="main"
Nov 04 00:00:51.250765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.250764    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerName="main"
Nov 04 00:00:51.251056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.251042    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="382849c4-a50a-43bd-80ea-1e1b147cb976" containerName="main"
Nov 04 00:00:51.256537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.256498    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.266600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.266564    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3a002b-openshift-default-dockercfg-qkqxj\""
Nov 04 00:00:51.266846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.266814    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:00:51.335256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.335213    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.335772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.335732    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336033    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336298    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336346    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336513    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336565    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf4j\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.336863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.336665    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.449878    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.449962    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf4j\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450065    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450101    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450278    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.450416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450326    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.451184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450467    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.451184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.450508    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.451184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.451046    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.475127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.470894    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.505460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.476340    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.505746 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.487051    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.505912 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.487287    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.506035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.501486    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.526563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.515894    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.555972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.555933    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.556851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.556825    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf4j\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\") pod \"e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") " pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:51.584108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:51.583750    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:52.054907 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:52.054868    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:00:52.055497 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:00:52.055467    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c76493c_8efd_42a4_90d6_cefa92b7788e.slice/crio-d34d206a0aaa868156b80ca09e572b4494435e1253876f94fc3337852d4878bf WatchSource:0}: Error finding container d34d206a0aaa868156b80ca09e572b4494435e1253876f94fc3337852d4878bf: Status 404 returned error can't find the container with id d34d206a0aaa868156b80ca09e572b4494435e1253876f94fc3337852d4878bf
Nov 04 00:00:52.058723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:52.058689    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:52.058811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:52.058774    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:52.058853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:52.058817    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:00:52.147284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:52.147249    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerStarted","Data":"d34d206a0aaa868156b80ca09e572b4494435e1253876f94fc3337852d4878bf"}
Nov 04 00:00:53.160476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:53.160438    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerStarted","Data":"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"}
Nov 04 00:00:53.264285 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:53.264249    6430 scope.go:122] "RemoveContainer" containerID="b3f4b5e213118c82a3ce45725bcbfbe2f93d65ec7f099827f16dc221fcd9d0d7"
Nov 04 00:00:53.584963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:53.584927    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:53.592658 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:53.592625    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:53.624149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:53.624090    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podStartSLOduration=2.624075719 podStartE2EDuration="2.624075719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:53.189895792 +0000 UTC m=+1869.080957738" watchObservedRunningTime="2026-11-04 00:00:53.624075719 +0000 UTC m=+1869.515137654"
Nov 04 00:00:54.188875 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.188841    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:54.190057 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.190033    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:00:54.542140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.542113    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:54.589532 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589490    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.589532 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.589811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589605    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.589811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589721    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.589811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589795    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.590150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589828    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.590150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589953    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.590150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.589987    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.590150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590040    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\") pod \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\" (UID: \"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb\") "
Nov 04 00:00:54.590374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590346    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:54.590665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590634    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:54.590665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:54.590853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:54.590853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590837    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:54.591036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.590978    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.591036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.591000    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.591036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.591012    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.591036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.591025    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.591249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.591038    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.594862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.594784    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "kube-api-access-h5l6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:54.594862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.594836    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:54.596717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.596682    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:54.596821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.596772    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy" pod "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" (UID: "19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:54.701771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.697979    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.701771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.698038    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.701771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.698051    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.701771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.698066    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5l6g\" (UniqueName: \"kubernetes.io/projected/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb-kube-api-access-h5l6g\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:54.940595 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.939674    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 04 00:00:54.996198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:54.996165    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 04 00:00:55.002464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002429    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002631 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002490    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002631 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002528    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002631 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002582    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002884    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.002972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.002927    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.003186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003081    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.003186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003130    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\") pod \"80a62d3d-85aa-4b79-9bcc-825453ab1333\" (UID: \"80a62d3d-85aa-4b79-9bcc-825453ab1333\") "
Nov 04 00:00:55.003568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003127    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.003568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003342    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.003568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003355    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.003877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003824    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.004000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.003886    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:55.004123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.004103    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.004207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.004124    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.004207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.004134    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a62d3d-85aa-4b79-9bcc-825453ab1333-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.004207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.004143    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.004207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.004152    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.006151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.006120    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.006424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.006356    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.006838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.006812    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:55.007274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.007239    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h" pod "80a62d3d-85aa-4b79-9bcc-825453ab1333" (UID: "80a62d3d-85aa-4b79-9bcc-825453ab1333"). InnerVolumeSpecName "kube-api-access-nz52h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.104744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.104704    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.104744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.104743    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.104948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.104772    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.104948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.104905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.104948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.104934    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.105134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105002    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.105134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105058    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.105241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105161    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.105241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105233    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.105365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105330    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.105365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105347    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\") pod \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\" (UID: \"17637757-1a0b-4ab1-807f-fcdaf2b8c352\") "
Nov 04 00:00:55.105552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105527    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.105625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105606    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.105677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.105647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:55.106097 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106078    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106097 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106099    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106111    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106126    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz52h\" (UniqueName: \"kubernetes.io/projected/80a62d3d-85aa-4b79-9bcc-825453ab1333-kube-api-access-nz52h\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106140    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a62d3d-85aa-4b79-9bcc-825453ab1333-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106152    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106164    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106177    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.106265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.106191    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.107322 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.107301    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.107595 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.107570    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:55.107595 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.107589    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.107730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.107572    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5" pod "17637757-1a0b-4ab1-807f-fcdaf2b8c352" (UID: "17637757-1a0b-4ab1-807f-fcdaf2b8c352"). InnerVolumeSpecName "kube-api-access-v2vw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.228409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.223181    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vw5\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-kube-api-access-v2vw5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.228409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.223226    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.228409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.223240    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.228409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.223254    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17637757-1a0b-4ab1-807f-fcdaf2b8c352-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.255427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.253127    6430 generic.go:356] "Generic (PLEG): container finished" podID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" containerID="6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2" exitCode=0
Nov 04 00:00:55.255427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.253221    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"
Nov 04 00:00:55.255427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.253332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" event={"ID":"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb","Type":"ContainerDied","Data":"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"}
Nov 04 00:00:55.255427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.253490    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf" event={"ID":"19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb","Type":"ContainerDied","Data":"9d1d1f25c29751491effab27cbd54fab67d0357cf6cba7a61b38c65866b222c3"}
Nov 04 00:00:55.255427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.253541    6430 scope.go:122] "RemoveContainer" containerID="6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"
Nov 04 00:00:55.258752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.257099    6430 generic.go:356] "Generic (PLEG): container finished" podID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerID="e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102" exitCode=0
Nov 04 00:00:55.258752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.257328    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"
Nov 04 00:00:55.258752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.257468    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" event={"ID":"17637757-1a0b-4ab1-807f-fcdaf2b8c352","Type":"ContainerDied","Data":"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"}
Nov 04 00:00:55.258752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.257522    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp" event={"ID":"17637757-1a0b-4ab1-807f-fcdaf2b8c352","Type":"ContainerDied","Data":"20ee1c5262f54cb77b897a5f3169d84a8d08f07cb6fcc7e75476414d4899a897"}
Nov 04 00:00:55.262734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.262105    6430 generic.go:356] "Generic (PLEG): container finished" podID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerID="840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75" exitCode=0
Nov 04 00:00:55.262734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.262142    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" event={"ID":"80a62d3d-85aa-4b79-9bcc-825453ab1333","Type":"ContainerDied","Data":"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"}
Nov 04 00:00:55.262734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.262203    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h" event={"ID":"80a62d3d-85aa-4b79-9bcc-825453ab1333","Type":"ContainerDied","Data":"7d6d7e797bc42a193062df584a2d7ea9038fae799da7ddd461cb7f49392db7af"}
Nov 04 00:00:55.262734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.262506    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"
Nov 04 00:00:55.271737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.271719    6430 scope.go:122] "RemoveContainer" containerID="6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"
Nov 04 00:00:55.272072 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:55.272048    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2\": container with ID starting with 6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2 not found: ID does not exist" containerID="6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"
Nov 04 00:00:55.272146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.272082    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2"} err="failed to get container status \"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2\": rpc error: code = NotFound desc = could not find container \"6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2\": container with ID starting with 6d06d7997a7c9ea3e7440b862fb59ee40f184f85779bc9d33864b0994d0c44c2 not found: ID does not exist"
Nov 04 00:00:55.272146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.272102    6430 scope.go:122] "RemoveContainer" containerID="e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"
Nov 04 00:00:55.282812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.282790    6430 scope.go:122] "RemoveContainer" containerID="e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"
Nov 04 00:00:55.283103 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:55.283081    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102\": container with ID starting with e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102 not found: ID does not exist" containerID="e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"
Nov 04 00:00:55.283146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.283114    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102"} err="failed to get container status \"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102\": rpc error: code = NotFound desc = could not find container \"e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102\": container with ID starting with e2c28ee87cc2fecbc24c14342fe030861b85378f39476fcb40828d7a27233102 not found: ID does not exist"
Nov 04 00:00:55.283146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.283132    6430 scope.go:122] "RemoveContainer" containerID="840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"
Nov 04 00:00:55.290122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.289756    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 04 00:00:55.294600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.294572    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-szzlp"]
Nov 04 00:00:55.296437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.296378    6430 scope.go:122] "RemoveContainer" containerID="840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"
Nov 04 00:00:55.296811 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:55.296787    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75\": container with ID starting with 840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75 not found: ID does not exist" containerID="840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"
Nov 04 00:00:55.296873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.296824    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75"} err="failed to get container status \"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75\": rpc error: code = NotFound desc = could not find container \"840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75\": container with ID starting with 840fc6ac92484a4296e258601270cdfd2eccea71c8baf3c8fc20e10c3ced9d75 not found: ID does not exist"
Nov 04 00:00:55.311965 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.310540    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:55.314878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.314849    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ba68e7-openshift-default-64b99d76c5-2zhhf"]
Nov 04 00:00:55.327383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.327348    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 04 00:00:55.330861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.330829    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-qnw7h"]
Nov 04 00:00:55.688913 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.688886    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752064    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752214    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752268    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752320    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752370    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752427    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752580    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.755359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.752635    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\") pod \"857b77fc-a180-4a57-b444-977daca6a832\" (UID: \"857b77fc-a180-4a57-b444-977daca6a832\") "
Nov 04 00:00:55.775432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.766771    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.775432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.767011    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.775432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.768027    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.775432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.768061    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.777182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.777147    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.780679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.780608    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:00:55.796029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.793216    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "kube-api-access-qf82h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.804715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.804678    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.810271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.810214    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:00:55.815139 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.815103    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:00:55.816632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.816558    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo" pod "857b77fc-a180-4a57-b444-977daca6a832" (UID: "857b77fc-a180-4a57-b444-977daca6a832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:00:55.869938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869887    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857b77fc-a180-4a57-b444-977daca6a832-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.869938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869927    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857b77fc-a180-4a57-b444-977daca6a832-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.869938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869936    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.869938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869945    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.870323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869956    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.870323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869964    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857b77fc-a180-4a57-b444-977daca6a832-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:55.870323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:55.869973    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf82h\" (UniqueName: \"kubernetes.io/projected/857b77fc-a180-4a57-b444-977daca6a832-kube-api-access-qf82h\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:00:56.234334 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.234300    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" path="/var/lib/kubelet/pods/17637757-1a0b-4ab1-807f-fcdaf2b8c352/volumes"
Nov 04 00:00:56.234933 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.234916    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" path="/var/lib/kubelet/pods/19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb/volumes"
Nov 04 00:00:56.235454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.235437    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" path="/var/lib/kubelet/pods/80a62d3d-85aa-4b79-9bcc-825453ab1333/volumes"
Nov 04 00:00:56.271481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.271445    6430 generic.go:356] "Generic (PLEG): container finished" podID="857b77fc-a180-4a57-b444-977daca6a832" containerID="124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2" exitCode=0
Nov 04 00:00:56.271670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.271495    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"
Nov 04 00:00:56.271670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.271532    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" event={"ID":"857b77fc-a180-4a57-b444-977daca6a832","Type":"ContainerDied","Data":"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"}
Nov 04 00:00:56.271670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.271586    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x" event={"ID":"857b77fc-a180-4a57-b444-977daca6a832","Type":"ContainerDied","Data":"b60a58d500f12fa9c857dfe93f2218fd93140a179d82ac47363a18130f09e2ad"}
Nov 04 00:00:56.271670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.271612    6430 scope.go:122] "RemoveContainer" containerID="124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"
Nov 04 00:00:56.282328 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.282299    6430 scope.go:122] "RemoveContainer" containerID="124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"
Nov 04 00:00:56.282638 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:00:56.282615    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2\": container with ID starting with 124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2 not found: ID does not exist" containerID="124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"
Nov 04 00:00:56.282716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.282647    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2"} err="failed to get container status \"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2\": rpc error: code = NotFound desc = could not find container \"124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2\": container with ID starting with 124c04489b6ae864267295e90b5fc8ede7b98bfb12b562ddfe196a2f1ea256b2 not found: ID does not exist"
Nov 04 00:00:56.308460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.308425    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 04 00:00:56.322378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:56.322343    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-n9b2x"]
Nov 04 00:00:58.234056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:58.234017    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857b77fc-a180-4a57-b444-977daca6a832" path="/var/lib/kubelet/pods/857b77fc-a180-4a57-b444-977daca6a832/volumes"
Nov 04 00:00:59.824046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:00:59.824013    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 04 00:01:00.005065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.005023    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:01:00.006250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006224    6430 cpu_manager.go:414] "containerMap: removing container" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy"
Nov 04 00:01:00.006355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006257    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy"
Nov 04 00:01:00.006355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006288    6430 cpu_manager.go:414] "containerMap: removing container" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy"
Nov 04 00:01:00.006355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006295    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy"
Nov 04 00:01:00.006355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006321    6430 cpu_manager.go:414] "containerMap: removing container" podUID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" containerName="istio-proxy"
Nov 04 00:01:00.006355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006330    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" containerName="istio-proxy"
Nov 04 00:01:00.006542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006404    6430 cpu_manager.go:414] "containerMap: removing container" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy"
Nov 04 00:01:00.006542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006413    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy"
Nov 04 00:01:00.006628 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006613    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="857b77fc-a180-4a57-b444-977daca6a832" containerName="istio-proxy"
Nov 04 00:01:00.006659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006637    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="17637757-1a0b-4ab1-807f-fcdaf2b8c352" containerName="istio-proxy"
Nov 04 00:01:00.006659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006652    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cd1d43-e19c-4b3a-ac86-8b6177cfe1fb" containerName="istio-proxy"
Nov 04 00:01:00.006720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.006669    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a62d3d-85aa-4b79-9bcc-825453ab1333" containerName="istio-proxy"
Nov 04 00:01:00.012720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.012693    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.025438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.022376    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2697a0-openshift-default-dockercfg-k6t4k\""
Nov 04 00:01:00.027933 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.027906    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:01:00.131895 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.131787    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132072 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.131964    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132072 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132005    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132215 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132204    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132254    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132317    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132355    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.132418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.132399    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsm56\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238542    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238683    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238769    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238825    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238853    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238926    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.238952    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsm56\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.241412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.239948    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.242095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.241694    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.245426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.244756    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.245426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.245017    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.283562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.270019    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.341693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.320485    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.341693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.321076    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.345686 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.345618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsm56\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.346351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.346302    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\") pod \"e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") " pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:00.634539 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:00.633120    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:01.695060 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:01.694867    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:01:01.819572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:01.819497    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main" containerID="cri-o://70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874" gracePeriod=45
Nov 04 00:01:01.845365 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:01:01.842726    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8008f1c5_c313_4781_8347_e155daa11c4b.slice/crio-9726840fa0dde821fc602cda2b43edf25be43077a0a9b225360e4d922a1fb032 WatchSource:0}: Error finding container 9726840fa0dde821fc602cda2b43edf25be43077a0a9b225360e4d922a1fb032: Status 404 returned error can't find the container with id 9726840fa0dde821fc602cda2b43edf25be43077a0a9b225360e4d922a1fb032
Nov 04 00:01:01.850436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:01.850283    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:01.850436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:01.850383    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:01.850594 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:01.850444    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:02.024707 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.021704    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 04 00:01:02.071412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.055681    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:01:02.071412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.056382    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": dial tcp 10.134.0.161:8000: connect: connection refused"
Nov 04 00:01:02.071412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.056508    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 04 00:01:02.083220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.083168    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:01:02.102647 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.102594    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 04 00:01:02.159624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.159559    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" podUID="369c472c-c589-48af-85c0-89320ae8ff57" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:01:02.782655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.782629    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 04 00:01:02.859081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.858973    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerStarted","Data":"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"}
Nov 04 00:01:02.859081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.859047    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerStarted","Data":"9726840fa0dde821fc602cda2b43edf25be43077a0a9b225360e4d922a1fb032"}
Nov 04 00:01:02.862659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.862503    6430 generic.go:356] "Generic (PLEG): container finished" podID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerID="70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874" exitCode=0
Nov 04 00:01:02.862659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.862552    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"
Nov 04 00:01:02.862659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.862589    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" event={"ID":"a0c21546-658b-4453-b8d6-462984e9cdf0","Type":"ContainerDied","Data":"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"}
Nov 04 00:01:02.862659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.862639    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9" event={"ID":"a0c21546-658b-4453-b8d6-462984e9cdf0","Type":"ContainerDied","Data":"abbfad33d708440bdaab6113ad750df498c809ff9704aaa49c699ac5bea0b2ef"}
Nov 04 00:01:02.862659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.862663    6430 scope.go:122] "RemoveContainer" containerID="70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"
Nov 04 00:01:02.870001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.869969    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.870137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.870037    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.870137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.870081    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\" (UniqueName: \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.870331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.870285    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.870483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.870434    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.870585 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.870569    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\") pod \"a0c21546-658b-4453-b8d6-462984e9cdf0\" (UID: \"a0c21546-658b-4453-b8d6-462984e9cdf0\") "
Nov 04 00:01:02.872535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.871508    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:02.872535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.871837    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:02.872535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.871878    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:02.876448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.876407    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:01:02.882613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.877863    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "kube-api-access-42kh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:02.882613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.878789    6430 scope.go:122] "RemoveContainer" containerID="70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"
Nov 04 00:01:02.882613 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:02.879129    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874\": container with ID starting with 70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874 not found: ID does not exist" containerID="70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"
Nov 04 00:01:02.882613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.879159    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874"} err="failed to get container status \"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874\": rpc error: code = NotFound desc = could not find container \"70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874\": container with ID starting with 70605310c4947e07d66ac9abd2317b7a4b75884151015bd6657ad39f4d069874 not found: ID does not exist"
Nov 04 00:01:02.882613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.879574    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm" pod "a0c21546-658b-4453-b8d6-462984e9cdf0" (UID: "a0c21546-658b-4453-b8d6-462984e9cdf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:02.913549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.913493    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podStartSLOduration=3.913479287 podStartE2EDuration="3.913479287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:02.909696734 +0000 UTC m=+1878.800758668" watchObservedRunningTime="2026-11-04 00:01:02.913479287 +0000 UTC m=+1878.804541355"
Nov 04 00:01:02.973024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.972984    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:02.973024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.973020    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c21546-658b-4453-b8d6-462984e9cdf0-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:02.973258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.973035    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:02.973258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.973062    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:02.973258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.973077    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42kh6\" (UniqueName: \"kubernetes.io/projected/a0c21546-658b-4453-b8d6-462984e9cdf0-kube-api-access-42kh6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:02.973258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:02.973092    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c21546-658b-4453-b8d6-462984e9cdf0-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:03.192654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.192602    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 04 00:01:03.201257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.201221    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a5948/auth-test-model-9a5948-kserve-78b5ddd9c-g9cj9"]
Nov 04 00:01:03.633764 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.633723    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:03.640752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.640716    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:03.871285 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.871250    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:03.872460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:03.872439    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:01:04.233658 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:04.233624    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" path="/var/lib/kubelet/pods/a0c21546-658b-4453-b8d6-462984e9cdf0/volumes"
Nov 04 00:01:04.833542 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:04.833502    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:01:06.388862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.388819    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 04 00:01:06.389482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.389112    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy" containerID="cri-o://35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3" gracePeriod=30
Nov 04 00:01:06.398234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.398204    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 04 00:01:06.398452 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.398429    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy" containerID="cri-o://ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76" gracePeriod=30
Nov 04 00:01:06.404875 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.404842    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 04 00:01:06.405107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:06.405078    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy" containerID="cri-o://73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050" gracePeriod=30
Nov 04 00:01:10.712423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:10.712374    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 04 00:01:11.709168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.709089    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 04 00:01:11.781946 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.781907    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 04 00:01:11.802048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802010    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802236 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802179    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802237    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802301    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802412    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802459    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.802546 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802494    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.802788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.802895 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802835    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.802957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802904    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.803009 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.802966    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.803070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.803028    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\") pod \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\" (UID: \"a7ba74fe-5fae-4505-ba57-83a44c9b884d\") "
Nov 04 00:01:11.803215 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.803193    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.803302 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.803282    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:11.803982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.803954    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.804108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.803985    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.804108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.804003    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.804108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.804017    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.804108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.804032    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.804948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.804923    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "kube-api-access-mq72j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:11.805199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.805173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:11.805288 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.805226    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.806549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.806518    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo" pod "a7ba74fe-5fae-4505-ba57-83a44c9b884d" (UID: "a7ba74fe-5fae-4505-ba57-83a44c9b884d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:11.808742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.808725    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 04 00:01:11.905536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905499    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.905536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905550    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.905812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905586    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.905812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905609    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.905812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905708    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.905812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905780    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905902    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.905933    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906013    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906047    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906086    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906167    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906212    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906441    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\") pod \"a21a9e85-6bb5-4036-9817-80c15e61b786\" (UID: \"a21a9e85-6bb5-4036-9817-80c15e61b786\") "
Nov 04 00:01:11.906916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906441    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.906916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906484    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906511    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\") pod \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\" (UID: \"288dd9f6-9dab-4a92-aa61-51c7a0e8e237\") "
Nov 04 00:01:11.906916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906746    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.907228 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.906969    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.907314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907270    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:11.907601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907520    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.907601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907544    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.907601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907559    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq72j\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-kube-api-access-mq72j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.907601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907576    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.907601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907590    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907603    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907618    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907630    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ba74fe-5fae-4505-ba57-83a44c9b884d-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907642    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907556    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.908035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.907851    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.908612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.908081    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.908612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.908131    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.908612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.908354    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:01:11.911674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.911650    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:11.911771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.911647    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.911954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.911939    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:11.912802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.912778    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:11.912901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.912808    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:01:11.912901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.912819    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh" pod "a21a9e85-6bb5-4036-9817-80c15e61b786" (UID: "a21a9e85-6bb5-4036-9817-80c15e61b786"). InnerVolumeSpecName "kube-api-access-9kvfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:11.912901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.912853    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:11.914022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:11.914001    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p" pod "288dd9f6-9dab-4a92-aa61-51c7a0e8e237" (UID: "288dd9f6-9dab-4a92-aa61-51c7a0e8e237"). InnerVolumeSpecName "kube-api-access-xk47p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009533    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009572    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk47p\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-kube-api-access-xk47p\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009590    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009605    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21a9e85-6bb5-4036-9817-80c15e61b786-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009619    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.009632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009634    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009650    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009664    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009678    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288dd9f6-9dab-4a92-aa61-51c7a0e8e237-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009693    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009707    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009720    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvfh\" (UniqueName: \"kubernetes.io/projected/a21a9e85-6bb5-4036-9817-80c15e61b786-kube-api-access-9kvfh\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.010108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.009735    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21a9e85-6bb5-4036-9817-80c15e61b786-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:12.043430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.034807    6430 generic.go:356] "Generic (PLEG): container finished" podID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerID="73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050" exitCode=0
Nov 04 00:01:12.043430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.034963    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" event={"ID":"288dd9f6-9dab-4a92-aa61-51c7a0e8e237","Type":"ContainerDied","Data":"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"}
Nov 04 00:01:12.043430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.035026    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2" event={"ID":"288dd9f6-9dab-4a92-aa61-51c7a0e8e237","Type":"ContainerDied","Data":"d9f22e572df3596dcba81620c5e3d92cc578f3a0cd96f484385ce3d5f10b200a"}
Nov 04 00:01:12.043430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.035057    6430 scope.go:122] "RemoveContainer" containerID="73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"
Nov 04 00:01:12.043430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.035223    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"
Nov 04 00:01:12.132523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.122857    6430 generic.go:356] "Generic (PLEG): container finished" podID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerID="ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76" exitCode=0
Nov 04 00:01:12.132523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.123105    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" event={"ID":"a21a9e85-6bb5-4036-9817-80c15e61b786","Type":"ContainerDied","Data":"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"}
Nov 04 00:01:12.132523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.123162    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr" event={"ID":"a21a9e85-6bb5-4036-9817-80c15e61b786","Type":"ContainerDied","Data":"ee43b82ba3630f8c2fa4b1609a0c4a8e288b05ec493c8d30507b06737c49143f"}
Nov 04 00:01:12.132523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.123263    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"
Nov 04 00:01:12.168027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.165750    6430 generic.go:356] "Generic (PLEG): container finished" podID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerID="35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3" exitCode=0
Nov 04 00:01:12.168027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.165910    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" event={"ID":"a7ba74fe-5fae-4505-ba57-83a44c9b884d","Type":"ContainerDied","Data":"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"}
Nov 04 00:01:12.168027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.165970    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl" event={"ID":"a7ba74fe-5fae-4505-ba57-83a44c9b884d","Type":"ContainerDied","Data":"f75d8e754f7165691f7e1656329573a283e5808caff24bca7f18deae5016571b"}
Nov 04 00:01:12.168027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.166069    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"
Nov 04 00:01:12.322641 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.321138    6430 scope.go:122] "RemoveContainer" containerID="73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"
Nov 04 00:01:12.381708 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:12.348646    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050\": container with ID starting with 73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050 not found: ID does not exist" containerID="73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"
Nov 04 00:01:12.381708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.348833    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050"} err="failed to get container status \"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050\": rpc error: code = NotFound desc = could not find container \"73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050\": container with ID starting with 73487e93a1c4b3c2ed5acff2db0b763b219f71da1198d5d3d30726ffe931c050 not found: ID does not exist"
Nov 04 00:01:12.381708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.348943    6430 scope.go:122] "RemoveContainer" containerID="ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"
Nov 04 00:01:12.427735 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.427708    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 04 00:01:12.433065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.433030    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-vsgbr"]
Nov 04 00:01:12.433675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.433657    6430 scope.go:122] "RemoveContainer" containerID="ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"
Nov 04 00:01:12.433985 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:12.433961    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76\": container with ID starting with ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76 not found: ID does not exist" containerID="ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"
Nov 04 00:01:12.434051 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.433992    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76"} err="failed to get container status \"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76\": rpc error: code = NotFound desc = could not find container \"ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76\": container with ID starting with ad240da6474e2b90dfc54e28865c620d334a92832434873d913240db165ddf76 not found: ID does not exist"
Nov 04 00:01:12.434051 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.434011    6430 scope.go:122] "RemoveContainer" containerID="35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"
Nov 04 00:01:12.440334 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.440309    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 04 00:01:12.443842 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.443819    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xcvv2"]
Nov 04 00:01:12.446123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.446107    6430 scope.go:122] "RemoveContainer" containerID="35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"
Nov 04 00:01:12.446501 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:12.446470    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3\": container with ID starting with 35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3 not found: ID does not exist" containerID="35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"
Nov 04 00:01:12.446571 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.446517    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3"} err="failed to get container status \"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3\": rpc error: code = NotFound desc = could not find container \"35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3\": container with ID starting with 35905a1058b5723fce60c92ce79784564247861b2e407dc8ea0bf63cd5f9efa3 not found: ID does not exist"
Nov 04 00:01:12.461936 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.461908    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 04 00:01:12.470662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.470637    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-s2jzl"]
Nov 04 00:01:12.595284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.595233    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:01:12.596170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596147    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596181    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596195    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596202    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596219    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596226    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596268    6430 cpu_manager.go:414] "containerMap: removing container" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy"
Nov 04 00:01:12.596277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596274    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy"
Nov 04 00:01:12.596705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596408    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" containerName="istio-proxy"
Nov 04 00:01:12.596705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596431    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" containerName="istio-proxy"
Nov 04 00:01:12.596705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596439    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" containerName="istio-proxy"
Nov 04 00:01:12.596705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.596447    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c21546-658b-4453-b8d6-462984e9cdf0" containerName="main"
Nov 04 00:01:12.602012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.601983    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.613819 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.613790    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:01:12.618054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.618029    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-27f8a1-serving-cert\""
Nov 04 00:01:12.665829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.665788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.666017 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.665910    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.666017 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.665953    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9rr\" (UniqueName: \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.666090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.666066    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.768411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.767735    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.768411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.768284    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.768411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.768370    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.768779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.768426    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9rr\" (UniqueName: \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.774415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.769653    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.774415 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:12.769769    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-serving-cert: secret "maas-api-e2e-autores-27f8a1-serving-cert" not found
Nov 04 00:01:12.774415 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:12.769839    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls podName:fbaa8ccb-6200-4e69-a47a-2dc7035b6220 nodeName:}" failed. No retries permitted until 2026-11-04 00:01:13.269821362 +0000 UTC m=+1889.160883277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls") pod "maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" (UID: "fbaa8ccb-6200-4e69-a47a-2dc7035b6220") : secret "maas-api-e2e-autores-27f8a1-serving-cert" not found
Nov 04 00:01:12.804056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.804019    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:12.814893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:12.808124    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9rr\" (UniqueName: \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:13.276465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:13.276420    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:13.278930 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:13.278903    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") pod \"maas-api-e2e-autores-27f8a1-7459698b4-n9xm6\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:13.529128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:13.527761    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:13.920061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:13.917423    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:01:13.921949 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:01:13.921905    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbaa8ccb_6200_4e69_a47a_2dc7035b6220.slice/crio-b5063d1c4ea4fd534cbb687c637b6ea777bf8c9370a004e0521a735a6757496c WatchSource:0}: Error finding container b5063d1c4ea4fd534cbb687c637b6ea777bf8c9370a004e0521a735a6757496c: Status 404 returned error can't find the container with id b5063d1c4ea4fd534cbb687c637b6ea777bf8c9370a004e0521a735a6757496c
Nov 04 00:01:13.923990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:13.923961    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:01:14.190562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:14.190478    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" event={"ID":"fbaa8ccb-6200-4e69-a47a-2dc7035b6220","Type":"ContainerStarted","Data":"b5063d1c4ea4fd534cbb687c637b6ea777bf8c9370a004e0521a735a6757496c"}
Nov 04 00:01:14.260416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:14.252244    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288dd9f6-9dab-4a92-aa61-51c7a0e8e237" path="/var/lib/kubelet/pods/288dd9f6-9dab-4a92-aa61-51c7a0e8e237/volumes"
Nov 04 00:01:14.260416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:14.253126    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21a9e85-6bb5-4036-9817-80c15e61b786" path="/var/lib/kubelet/pods/a21a9e85-6bb5-4036-9817-80c15e61b786/volumes"
Nov 04 00:01:14.298440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:14.293523    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ba74fe-5fae-4505-ba57-83a44c9b884d" path="/var/lib/kubelet/pods/a7ba74fe-5fae-4505-ba57-83a44c9b884d/volumes"
Nov 04 00:01:14.847343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:14.847294    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" podUID="369c472c-c589-48af-85c0-89320ae8ff57" containerName="main" containerID="cri-o://5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab" gracePeriod=45
Nov 04 00:01:15.136722 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.136692    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 04 00:01:15.204509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.204446    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" event={"ID":"fbaa8ccb-6200-4e69-a47a-2dc7035b6220","Type":"ContainerStarted","Data":"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"}
Nov 04 00:01:15.206429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.206354    6430 generic.go:356] "Generic (PLEG): container finished" podID="369c472c-c589-48af-85c0-89320ae8ff57" containerID="5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab" exitCode=0
Nov 04 00:01:15.206649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.206447    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" event={"ID":"369c472c-c589-48af-85c0-89320ae8ff57","Type":"ContainerDied","Data":"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"}
Nov 04 00:01:15.206649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.206480    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf" event={"ID":"369c472c-c589-48af-85c0-89320ae8ff57","Type":"ContainerDied","Data":"70ec204cd85241b1c4a0e36ddac9f80dfed00a8c05c7e5f960439afdceaa48cf"}
Nov 04 00:01:15.206649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.206502    6430 scope.go:122] "RemoveContainer" containerID="5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"
Nov 04 00:01:15.207145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.207119    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"
Nov 04 00:01:15.218107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.218082    6430 scope.go:122] "RemoveContainer" containerID="5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"
Nov 04 00:01:15.218423 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:15.218372    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab\": container with ID starting with 5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab not found: ID does not exist" containerID="5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"
Nov 04 00:01:15.218523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.218418    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab"} err="failed to get container status \"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab\": rpc error: code = NotFound desc = could not find container \"5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab\": container with ID starting with 5f667c9b631bcbdd306ef017f54e04c9eed5477cadbee57b5ccdb68b322328ab not found: ID does not exist"
Nov 04 00:01:15.230938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.230886    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" podStartSLOduration=2.7554521 podStartE2EDuration="3.230872009s" totalImagesPullingTime="475.419909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:15.227755212 +0000 UTC m=+1891.118817142" watchObservedRunningTime="2026-11-04 00:01:15.230872009 +0000 UTC m=+1891.121933948"
Nov 04 00:01:15.242726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.242698    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.242893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.242746    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.242893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.242838    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\" (UniqueName: \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.243037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243015    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:15.243037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243029    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:15.243141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243090    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.243193 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243171    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.243246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243205    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\" (UniqueName: \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\") pod \"369c472c-c589-48af-85c0-89320ae8ff57\" (UID: \"369c472c-c589-48af-85c0-89320ae8ff57\") "
Nov 04 00:01:15.243328 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243308    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:15.243983 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243956    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.244104 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.243984    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.244104 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.244001    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.245558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.245531    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:01:15.245772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.245751    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:15.245909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.245892    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2" pod "369c472c-c589-48af-85c0-89320ae8ff57" (UID: "369c472c-c589-48af-85c0-89320ae8ff57"). InnerVolumeSpecName "kube-api-access-b8zp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:15.346013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.345967    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369c472c-c589-48af-85c0-89320ae8ff57-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.346013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.346000    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zp2\" (UniqueName: \"kubernetes.io/projected/369c472c-c589-48af-85c0-89320ae8ff57-kube-api-access-b8zp2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.346221 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.346016    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369c472c-c589-48af-85c0-89320ae8ff57-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:15.550223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.550179    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 04 00:01:15.553871 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:15.553838    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2341ca/auth-test-model-2341ca-kserve-5866cbc745-z4crf"]
Nov 04 00:01:16.215012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:16.214972    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:16.233287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:16.233256    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369c472c-c589-48af-85c0-89320ae8ff57" path="/var/lib/kubelet/pods/369c472c-c589-48af-85c0-89320ae8ff57/volumes"
Nov 04 00:01:22.233202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:22.233170    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:01:25.736537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:25.736495    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" podUID="6385f9dc-71ad-456b-a580-5693f3cf0770" containerName="main" containerID="cri-o://1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a" gracePeriod=45
Nov 04 00:01:26.028448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.028417    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 04 00:01:26.094942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.094907    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.095181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095164    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095210    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095380    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\" (UniqueName: \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095614    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\" (UniqueName: \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.095784    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\") pod \"6385f9dc-71ad-456b-a580-5693f3cf0770\" (UID: \"6385f9dc-71ad-456b-a580-5693f3cf0770\") "
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.096261    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.096572    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.096775    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.096795    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.096834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.096808    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.097998 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.097977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:01:26.099256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.099235    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:01:26.099856 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.099824    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx" pod "6385f9dc-71ad-456b-a580-5693f3cf0770" (UID: "6385f9dc-71ad-456b-a580-5693f3cf0770"). InnerVolumeSpecName "kube-api-access-d4bjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:01:26.199085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.199046    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6385f9dc-71ad-456b-a580-5693f3cf0770-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.199085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.199085    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4bjx\" (UniqueName: \"kubernetes.io/projected/6385f9dc-71ad-456b-a580-5693f3cf0770-kube-api-access-d4bjx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.199301 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.199103    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6385f9dc-71ad-456b-a580-5693f3cf0770-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:01:26.349736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.349700    6430 generic.go:356] "Generic (PLEG): container finished" podID="6385f9dc-71ad-456b-a580-5693f3cf0770" containerID="1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a" exitCode=0
Nov 04 00:01:26.349736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.349740    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"
Nov 04 00:01:26.349962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.349780    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" event={"ID":"6385f9dc-71ad-456b-a580-5693f3cf0770","Type":"ContainerDied","Data":"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"}
Nov 04 00:01:26.349962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.349823    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9" event={"ID":"6385f9dc-71ad-456b-a580-5693f3cf0770","Type":"ContainerDied","Data":"21b0ea3aee403454eae9627b1a5953a165b955fef54c4391df8863ba14255189"}
Nov 04 00:01:26.349962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.349840    6430 scope.go:122] "RemoveContainer" containerID="1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"
Nov 04 00:01:26.360546 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.360522    6430 scope.go:122] "RemoveContainer" containerID="1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"
Nov 04 00:01:26.360858 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:26.360836    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a\": container with ID starting with 1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a not found: ID does not exist" containerID="1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"
Nov 04 00:01:26.360935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.360872    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a"} err="failed to get container status \"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a\": rpc error: code = NotFound desc = could not find container \"1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a\": container with ID starting with 1827e376ded5e43ec1ad40982c72017a5ff0f599422dc2693df1e32b7326843a not found: ID does not exist"
Nov 04 00:01:26.376299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.376262    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 04 00:01:26.380563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.380531    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2b1f66/auth-test-model-2b1f66-kserve-b8795c965-p5qn9"]
Nov 04 00:01:26.822134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.822096    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:01:26.823045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823019    6430 cpu_manager.go:414] "containerMap: removing container" podUID="369c472c-c589-48af-85c0-89320ae8ff57" containerName="main"
Nov 04 00:01:26.823045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823042    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369c472c-c589-48af-85c0-89320ae8ff57" containerName="main"
Nov 04 00:01:26.823215 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823085    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6385f9dc-71ad-456b-a580-5693f3cf0770" containerName="main"
Nov 04 00:01:26.823215 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823090    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6385f9dc-71ad-456b-a580-5693f3cf0770" containerName="main"
Nov 04 00:01:26.823283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823219    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6385f9dc-71ad-456b-a580-5693f3cf0770" containerName="main"
Nov 04 00:01:26.823283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.823235    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="369c472c-c589-48af-85c0-89320ae8ff57" containerName="main"
Nov 04 00:01:26.828229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.828208    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:26.840584 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.840558    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2697a0-serving-cert\""
Nov 04 00:01:26.851512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.851480    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:01:26.909170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.909133    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:26.909411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.909297    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2vp\" (UniqueName: \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:26.909411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.909320    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:26.909411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:26.909347    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.011326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.011285    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2vp\" (UniqueName: \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.011326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.011330    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.011637 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.011357    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.011637 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:27.011502    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-serving-cert: secret "maas-api-e2e-extmod-b-2697a0-serving-cert" not found
Nov 04 00:01:27.011637 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.011544    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.011637 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:27.011586    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls podName:2c13f4ca-345b-4842-bd79-f341aa0d5efa nodeName:}" failed. No retries permitted until 2026-11-04 00:01:27.51156431 +0000 UTC m=+1903.402626223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls") pod "maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" (UID: "2c13f4ca-345b-4842-bd79-f341aa0d5efa") : secret "maas-api-e2e-extmod-b-2697a0-serving-cert" not found
Nov 04 00:01:27.012230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.012208    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.014029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.014006    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.026890 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.026856    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2vp\" (UniqueName: \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.231122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.231026    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:01:27.236118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.236092    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.271913 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.271869    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:01:27.286797 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.286759    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-18f5c0-dockercfg-mfhmh\""
Nov 04 00:01:27.315703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.315657    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.315963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.315937    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hfs\" (UniqueName: \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.339349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.339317    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-18f5c0\""
Nov 04 00:01:27.430416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.421462    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hfs\" (UniqueName: \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.430416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.421751    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.430416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.422951    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.475061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.474970    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hfs\" (UniqueName: \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\") pod \"payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") " pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.526466 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.525902    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.535436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.532479    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:27.574408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.573690    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:27.744720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:27.744013    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:28.022579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.022543    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:01:28.095194 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:01:28.090092    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1adfc7a_5459_474a_83ae_25b93702a68f.slice/crio-67c0e11efdc1701fdf6deccd4be0ca8744660979822a1f66e4a67d6903ace3e9 WatchSource:0}: Error finding container 67c0e11efdc1701fdf6deccd4be0ca8744660979822a1f66e4a67d6903ace3e9: Status 404 returned error can't find the container with id 67c0e11efdc1701fdf6deccd4be0ca8744660979822a1f66e4a67d6903ace3e9
Nov 04 00:01:28.117173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.117090    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:01:28.234037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.234010    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6385f9dc-71ad-456b-a580-5693f3cf0770" path="/var/lib/kubelet/pods/6385f9dc-71ad-456b-a580-5693f3cf0770/volumes"
Nov 04 00:01:28.366851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.366814    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" event={"ID":"2c13f4ca-345b-4842-bd79-f341aa0d5efa","Type":"ContainerStarted","Data":"2f0776f6c75f2e586d7945166eae4752edc2923e3799113a937108c266b78b3a"}
Nov 04 00:01:28.368569 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.368488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" event={"ID":"c1adfc7a-5459-474a-83ae-25b93702a68f","Type":"ContainerStarted","Data":"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"}
Nov 04 00:01:28.368569 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.368539    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" event={"ID":"c1adfc7a-5459-474a-83ae-25b93702a68f","Type":"ContainerStarted","Data":"67c0e11efdc1701fdf6deccd4be0ca8744660979822a1f66e4a67d6903ace3e9"}
Nov 04 00:01:28.368831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:28.368712    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:29.378611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:29.378566    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" event={"ID":"2c13f4ca-345b-4842-bd79-f341aa0d5efa","Type":"ContainerStarted","Data":"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"}
Nov 04 00:01:29.379046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:29.378893    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:29.404373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:29.404315    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" podStartSLOduration=2.801248592 podStartE2EDuration="3.40429896s" totalImagesPullingTime="603.050368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:29.401351997 +0000 UTC m=+1905.292413930" watchObservedRunningTime="2026-11-04 00:01:29.40429896 +0000 UTC m=+1905.295360895"
Nov 04 00:01:29.405286 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:29.405240    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" podStartSLOduration=2.405226436 podStartE2EDuration="2.405226436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:28.432192645 +0000 UTC m=+1904.323254581" watchObservedRunningTime="2026-11-04 00:01:29.405226436 +0000 UTC m=+1905.296288370"
Nov 04 00:01:33.929336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:33.929296    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:01:35.004585 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:01:35.004539    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:01:35.395307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:35.394574    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:01:39.383127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.383093    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:01:39.959429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.959278    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:01:39.965533 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.965505    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:01:39.965695 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.965608    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:39.976709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.976678    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-27f8a1\"/\"autores-model-27f8a1-kserve-self-signed-certs\""
Nov 04 00:01:39.978424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.978382    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-27f8a1\"/\"default-dockercfg-5rgkp\""
Nov 04 00:01:39.981383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.981355    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-27f8a1\"/\"openshift-service-ca.crt\""
Nov 04 00:01:39.981508 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:39.981411    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-27f8a1\"/\"kube-root-ca.crt\""
Nov 04 00:01:40.071249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.071476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071270    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.071549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071492    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.071684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071661    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.071815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071790    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.071922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.071904    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mxd\" (UniqueName: \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.173557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.173507    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.182338 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.182301    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.183284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.183257    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.183892 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.183870    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mxd\" (UniqueName: \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.184116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.184097    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.184257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.184241    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.185217 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.174274    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.195048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.183757    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.195766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.195729    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.262182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.262102    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.286410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.277638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mxd\" (UniqueName: \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.303127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.303023    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\") pod \"autores-model-27f8a1-kserve-5cfcc589bf-tb8xl\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") " pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.580378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.580178    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:40.999170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:40.999137    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:01:40.999819 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:01:40.999790    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30ad8c6_6ea3_476f_85a0_dc47867c24ad.slice/crio-529f8c1140b7ff8e871ca824a04d350c1566908eb0c35cc0d57d31c33cf54260 WatchSource:0}: Error finding container 529f8c1140b7ff8e871ca824a04d350c1566908eb0c35cc0d57d31c33cf54260: Status 404 returned error can't find the container with id 529f8c1140b7ff8e871ca824a04d350c1566908eb0c35cc0d57d31c33cf54260
Nov 04 00:01:41.701869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:41.701776    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" event={"ID":"b30ad8c6-6ea3-476f-85a0-dc47867c24ad","Type":"ContainerStarted","Data":"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"}
Nov 04 00:01:41.701869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:41.701825    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" event={"ID":"b30ad8c6-6ea3-476f-85a0-dc47867c24ad","Type":"ContainerStarted","Data":"529f8c1140b7ff8e871ca824a04d350c1566908eb0c35cc0d57d31c33cf54260"}
Nov 04 00:01:41.727062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:41.727004    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" podStartSLOduration=2.726988993 podStartE2EDuration="2.726988993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:41.723772072 +0000 UTC m=+1917.614834030" watchObservedRunningTime="2026-11-04 00:01:41.726988993 +0000 UTC m=+1917.618050924"
Nov 04 00:01:42.580487 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:42.580447    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:45.744813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.744776    6430 generic.go:356] "Generic (PLEG): container finished" podID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerID="5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd" exitCode=0
Nov 04 00:01:45.745237 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.744824    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerDied","Data":"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"}
Nov 04 00:01:45.745237 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.745212    6430 scope.go:122] "RemoveContainer" containerID="5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"
Nov 04 00:01:45.746935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.746905    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:45.747013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.746978    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:45.747054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.747013    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:01:45.991062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.990972    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:01:45.994997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:45.994970    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.007512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.007476    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-18f5c0-serving-cert\""
Nov 04 00:01:46.014086 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.014051    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:01:46.081095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.081056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.081288 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.081203    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdh9\" (UniqueName: \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.081469 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.081436    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.081564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.081535    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.188174 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.188111    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.191638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.191591    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.192716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.192691    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.196855 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.196825    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdh9\" (UniqueName: \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.197592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.191438    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.258216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.258012    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.262451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.259383    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.289665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.288938    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdh9\" (UniqueName: \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\") pod \"maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.326989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.326951    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:46.630421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.626695    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:01:46.863772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.863733    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerStarted","Data":"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"}
Nov 04 00:01:46.879422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:46.875415    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" event={"ID":"fa8ec593-5a98-47cb-addd-eaf90029098d","Type":"ContainerStarted","Data":"8bfcbdc0f2a91e47d13babb299a7fcbb8b391b08377be17087460c863e0b3d60"}
Nov 04 00:01:47.601412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.598564    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:01:47.601412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.598965    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:47.601412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.599104    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:47.884339 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.884243    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" event={"ID":"fa8ec593-5a98-47cb-addd-eaf90029098d","Type":"ContainerStarted","Data":"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"}
Nov 04 00:01:47.884781 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.884346    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:47.933941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:47.933873    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" podStartSLOduration=2.4669316820000002 podStartE2EDuration="2.933850742s" totalImagesPullingTime="466.91906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:47.921752229 +0000 UTC m=+1923.812814164" watchObservedRunningTime="2026-11-04 00:01:47.933850742 +0000 UTC m=+1923.824912677"
Nov 04 00:01:48.585164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:48.585127    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:48.585424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:48.585188    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:49.585784 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:49.585743    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:49.586206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:49.585808    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:50.585413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:50.582965    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:50.607412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:50.602628    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:50.607412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:50.602725    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:50.790007 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:50.789973    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:50.993645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:50.993551    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:01:51.584838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:51.584796    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:51.585076 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:51.585055    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:52.585874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:52.585834    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 04 00:01:52.586410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:52.585936    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 04 00:01:53.590979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:53.590948    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:01:53.591463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:53.591421    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:01:53.592238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:53.592215    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:01:53.937700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:53.937661    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:01:55.449747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:01:55.449706    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:02:04.833693 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:04.833654    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:02:07.520113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:07.520072    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:02:10.484030 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.483926    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" podUID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" containerName="main" containerID="cri-o://0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990" gracePeriod=45
Nov 04 00:02:10.670974 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:10.670940    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30ad8c6_6ea3_476f_85a0_dc47867c24ad.slice/crio-0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:02:10.871286 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.871258    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940307    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940436    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940555    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\" (UniqueName: \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940588    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940657    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.942489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.940691    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\") pod \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\" (UID: \"b30ad8c6-6ea3-476f-85a0-dc47867c24ad\") "
Nov 04 00:02:10.943306 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.943276    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:10.983433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.943695    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:10.991559 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.990910    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:10.992470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.992216    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "kube-api-access-z8mxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:02:10.993582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.993542    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:10.994102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:10.994074    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs" pod "b30ad8c6-6ea3-476f-85a0-dc47867c24ad" (UID: "b30ad8c6-6ea3-476f-85a0-dc47867c24ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:02:11.043034 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.042996    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8mxd\" (UniqueName: \"kubernetes.io/projected/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-kube-api-access-z8mxd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.043248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.043050    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.043248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.043061    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.043248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.043072    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.043248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.043081    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.043248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.043089    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30ad8c6-6ea3-476f-85a0-dc47867c24ad-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:11.379362 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.379328    6430 generic.go:356] "Generic (PLEG): container finished" podID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" containerID="0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990" exitCode=0
Nov 04 00:02:11.379632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.379377    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" event={"ID":"b30ad8c6-6ea3-476f-85a0-dc47867c24ad","Type":"ContainerDied","Data":"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"}
Nov 04 00:02:11.379632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.379436    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl" event={"ID":"b30ad8c6-6ea3-476f-85a0-dc47867c24ad","Type":"ContainerDied","Data":"529f8c1140b7ff8e871ca824a04d350c1566908eb0c35cc0d57d31c33cf54260"}
Nov 04 00:02:11.379632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.379453    6430 scope.go:122] "RemoveContainer" containerID="0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"
Nov 04 00:02:11.379632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.379579    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"
Nov 04 00:02:11.393990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.393968    6430 scope.go:122] "RemoveContainer" containerID="0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"
Nov 04 00:02:11.394750 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:11.394721    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990\": container with ID starting with 0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990 not found: ID does not exist" containerID="0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"
Nov 04 00:02:11.394804 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.394764    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990"} err="failed to get container status \"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990\": rpc error: code = NotFound desc = could not find container \"0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990\": container with ID starting with 0c766c8b69a1f9223d0a58d21db936998a46b4eecdb2609b77dcf69b7b598990 not found: ID does not exist"
Nov 04 00:02:11.434411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.434211    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:02:11.437874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.437841    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-27f8a1/autores-model-27f8a1-kserve-5cfcc589bf-tb8xl"]
Nov 04 00:02:11.830413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.826439    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:02:11.830413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.827701    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" containerName="main"
Nov 04 00:02:11.830413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.827729    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" containerName="main"
Nov 04 00:02:11.830413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.827952    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" containerName="main"
Nov 04 00:02:11.834422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.834207    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.838991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.838955    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:02:11.848661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.848630    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8895cb-serving-cert\""
Nov 04 00:02:11.876654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.876608    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.877209 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.877171    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.877314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.877257    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.877461 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.877437    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvc4p\" (UniqueName: \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.978942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.978899    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.979158 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.979027    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvc4p\" (UniqueName: \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.979158 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:11.979049    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-serving-cert: secret "maas-api-e2e-extmod-a-8895cb-serving-cert" not found
Nov 04 00:02:11.979158 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:11.979135    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls podName:52fdf076-ffef-4a57-9630-4f6e0f5a8ebe nodeName:}" failed. No retries permitted until 2026-11-04 00:02:12.479114826 +0000 UTC m=+1948.370176738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls") pod "maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" (UID: "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe") : secret "maas-api-e2e-extmod-a-8895cb-serving-cert" not found
Nov 04 00:02:11.979357 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.979172    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.979458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.979424    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:11.980235 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:11.980208    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:12.010415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.009696    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:12.023413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.017499    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvc4p\" (UniqueName: \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:12.242214 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.242122    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30ad8c6-6ea3-476f-85a0-dc47867c24ad" path="/var/lib/kubelet/pods/b30ad8c6-6ea3-476f-85a0-dc47867c24ad/volumes"
Nov 04 00:02:12.507186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.507077    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:12.546413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.538437    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:12.758509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:12.754916    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:13.246411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.245535    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:02:13.389484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.372655    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:13.451677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.451643    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:13.451916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.451768    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.464067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.464037    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-18f5c0\"/\"openshift-service-ca.crt\""
Nov 04 00:02:13.466762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.465178    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" event={"ID":"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe","Type":"ContainerStarted","Data":"2d90d20ef628c7ff1bc79a52980c3023fe86bc2456a679ab27205d4a25c264f9"}
Nov 04 00:02:13.466762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.466365    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-18f5c0\"/\"kube-root-ca.crt\""
Nov 04 00:02:13.466762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.466374    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-18f5c0\"/\"explicit-model-18f5c0-kserve-self-signed-certs\""
Nov 04 00:02:13.466762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.466472    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-18f5c0\"/\"default-dockercfg-g6pww\""
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.594956    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.595038    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.595071    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.595192    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.595224    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bldk\" (UniqueName: \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.596411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.595274    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.698550    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.698610    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bldk\" (UniqueName: \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.698661    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.698949    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.699019    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.699043    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.699638    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.699997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.699947    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.713406 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.707149    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.731090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.714991    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.731090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.715502    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.737407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.732518    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bldk\" (UniqueName: \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\") pod \"explicit-model-18f5c0-kserve-7cf79d8489-cwsfs\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") " pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:13.777602 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:13.777574    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:14.172407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.163623    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:14.213810 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:02:14.203006    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9079aa9a_c107_4bf5_b1b4_44ee6435fee1.slice/crio-55e1756f6952131d550d9d299555d96787474e40ca38bc18fa79ea6021da600e WatchSource:0}: Error finding container 55e1756f6952131d550d9d299555d96787474e40ca38bc18fa79ea6021da600e: Status 404 returned error can't find the container with id 55e1756f6952131d550d9d299555d96787474e40ca38bc18fa79ea6021da600e
Nov 04 00:02:14.474348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.474241    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" event={"ID":"9079aa9a-c107-4bf5-b1b4-44ee6435fee1","Type":"ContainerStarted","Data":"26644a1e5b06b7270debe03e2b8f51b0ff56fd2d34c04782151a66687c5a26ec"}
Nov 04 00:02:14.474348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.474299    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" event={"ID":"9079aa9a-c107-4bf5-b1b4-44ee6435fee1","Type":"ContainerStarted","Data":"55e1756f6952131d550d9d299555d96787474e40ca38bc18fa79ea6021da600e"}
Nov 04 00:02:14.476292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.476249    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" event={"ID":"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe","Type":"ContainerStarted","Data":"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"}
Nov 04 00:02:14.476468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.476364    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:14.505856 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.505798    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" podStartSLOduration=1.505780139 podStartE2EDuration="1.505780139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:14.503768067 +0000 UTC m=+1950.394830002" watchObservedRunningTime="2026-11-04 00:02:14.505780139 +0000 UTC m=+1950.396842103"
Nov 04 00:02:14.547976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.547915    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" podStartSLOduration=2.957767909 podStartE2EDuration="3.547899471s" totalImagesPullingTime="590.131562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:14.545656373 +0000 UTC m=+1950.436718309" watchObservedRunningTime="2026-11-04 00:02:14.547899471 +0000 UTC m=+1950.438961405"
Nov 04 00:02:14.778430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:14.778319    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:19.803198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:19.802601    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": read tcp 10.134.0.2:46640->10.134.0.173:15021: read: connection reset by peer" start-of-body=
Nov 04 00:02:19.803198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:19.802694    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": read tcp 10.134.0.2:46640->10.134.0.173:15021: read: connection reset by peer"
Nov 04 00:02:20.488967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.488934    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:02:20.543196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.543162    6430 generic.go:356] "Generic (PLEG): container finished" podID="8008f1c5-c313-4781-8347-e155daa11c4b" containerID="4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a" exitCode=0
Nov 04 00:02:20.543413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.543243    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerDied","Data":"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"}
Nov 04 00:02:20.543858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.543830    6430 scope.go:122] "RemoveContainer" containerID="4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"
Nov 04 00:02:20.546031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.546000    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:20.546121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.546074    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:20.546121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:20.546107    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:21.553364 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:21.553329    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerStarted","Data":"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"}
Nov 04 00:02:21.633691 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:21.633654    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:02:21.633974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:21.633941    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:21.634052 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:21.634026    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:22.633966 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:22.633930    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:22.634521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:22.634493    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:23.642229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:23.642190    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:23.642823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:23.642514    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:23.778892 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:23.778849    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:23.792071 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:23.792042    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:24.249662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:24.249622    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:24.596843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:24.596809    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:24.634150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:24.634097    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:24.634383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:24.634183    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:24.666679 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:24.666571    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:24.666679 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:24.666670    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:25.166647276 +0000 UTC m=+1961.057709204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:25.174777 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:25.174737    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:25.175100 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:25.175082    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:26.175059804 +0000 UTC m=+1962.066121720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:25.634098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:25.634052    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:25.634436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:25.634407    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:26.193561 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:26.193524    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:26.194052 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:26.194032    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:28.194014523 +0000 UTC m=+1964.085076440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:26.634521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:26.634480    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 04 00:02:26.634716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:26.634536    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 04 00:02:27.638885 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:27.638846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:02:27.639307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:27.639259    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:02:27.640024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:27.640000    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:02:28.237464 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:28.237421    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:28.237644 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:28.237498    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:32.237478876 +0000 UTC m=+1968.128540815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:32.261549 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:32.261514    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:32.262077 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:32.261608    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:40.261590063 +0000 UTC m=+1976.152651983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:34.834772 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:34.834732    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:02:40.262720 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:40.262680    6430 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-self-signed-certs: secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:40.263163 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:40.262763    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs podName:9079aa9a-c107-4bf5-b1b4-44ee6435fee1 nodeName:}" failed. No retries permitted until 2026-11-04 00:02:56.26274784 +0000 UTC m=+1992.153809758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs") pod "explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1") : secret "explicit-model-18f5c0-kserve-self-signed-certs" not found
Nov 04 00:02:40.638334 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:40.638289    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" podUID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" containerName="main" containerID="cri-o://26644a1e5b06b7270debe03e2b8f51b0ff56fd2d34c04782151a66687c5a26ec" gracePeriod=45
Nov 04 00:02:40.956432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:40.956373    6430 generic.go:356] "Generic (PLEG): container finished" podID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" containerID="26644a1e5b06b7270debe03e2b8f51b0ff56fd2d34c04782151a66687c5a26ec" exitCode=0
Nov 04 00:02:40.956957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:40.956427    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" event={"ID":"9079aa9a-c107-4bf5-b1b4-44ee6435fee1","Type":"ContainerDied","Data":"26644a1e5b06b7270debe03e2b8f51b0ff56fd2d34c04782151a66687c5a26ec"}
Nov 04 00:02:41.141268 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.141235    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:41.215437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.215321    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.221016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.216538    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:41.221520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.221490    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.222251 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.222220    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.222460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.222341    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.222569 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.222472    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\" (UniqueName: \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.222569 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.222513    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\") pod \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\" (UID: \"9079aa9a-c107-4bf5-b1b4-44ee6435fee1\") "
Nov 04 00:02:41.223377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.223335    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.223865 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.223821    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:41.225338 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.225302    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:41.225951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.225918    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "kube-api-access-6bldk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:02:41.226173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.226135    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:02:41.232223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.232191    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs" pod "9079aa9a-c107-4bf5-b1b4-44ee6435fee1" (UID: "9079aa9a-c107-4bf5-b1b4-44ee6435fee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:02:41.324436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.324383    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.324436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.324430    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.324436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.324442    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.324927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.324452    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bldk\" (UniqueName: \"kubernetes.io/projected/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-kube-api-access-6bldk\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.324927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.324461    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9079aa9a-c107-4bf5-b1b4-44ee6435fee1-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:41.950410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.947513    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:02:41.950410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.948693    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" containerName="main"
Nov 04 00:02:41.950410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.948720    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" containerName="main"
Nov 04 00:02:41.950410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.948995    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" containerName="main"
Nov 04 00:02:41.961410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.958485    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.969716    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.969955    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-hsvcw\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.970010    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.970050    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.970097    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.970131    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-261bd6139428\""
Nov 04 00:02:41.981411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:41.979369    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:02:42.004727 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.004368    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs" event={"ID":"9079aa9a-c107-4bf5-b1b4-44ee6435fee1","Type":"ContainerDied","Data":"55e1756f6952131d550d9d299555d96787474e40ca38bc18fa79ea6021da600e"}
Nov 04 00:02:42.004727 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.004718    6430 scope.go:122] "RemoveContainer" containerID="26644a1e5b06b7270debe03e2b8f51b0ff56fd2d34c04782151a66687c5a26ec"
Nov 04 00:02:42.005007 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.004898    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"
Nov 04 00:02:42.065880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.065835    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:42.068145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.068117    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.068341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.068312    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.068428 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.068367    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.068628 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.068612    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwhd\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.069543 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.069516    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-18f5c0/explicit-model-18f5c0-kserve-7cf79d8489-cwsfs"]
Nov 04 00:02:42.169944 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.169905    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwhd\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.170154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.170016    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.170269 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.170176    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.170269 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.170211    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.170931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.170904    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.171145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.171120    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.172827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.172806    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.183191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.183162    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwhd\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.313994 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.311695    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:42.314747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.314704    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9079aa9a-c107-4bf5-b1b4-44ee6435fee1" path="/var/lib/kubelet/pods/9079aa9a-c107-4bf5-b1b4-44ee6435fee1/volumes"
Nov 04 00:02:42.738601 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:02:42.738568    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be52f8e_593b_43d3_9f1e_9fe715d2851c.slice/crio-ed6c7a18a3d6454758827fb6eddadffda1c40c1028944e41c45e112eee327988 WatchSource:0}: Error finding container ed6c7a18a3d6454758827fb6eddadffda1c40c1028944e41c45e112eee327988: Status 404 returned error can't find the container with id ed6c7a18a3d6454758827fb6eddadffda1c40c1028944e41c45e112eee327988
Nov 04 00:02:42.739610 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:42.739579    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:02:43.015008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:43.014910    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" event={"ID":"9be52f8e-593b-43d3-9f1e-9fe715d2851c","Type":"ContainerStarted","Data":"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"}
Nov 04 00:02:43.015008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:43.014970    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" event={"ID":"9be52f8e-593b-43d3-9f1e-9fe715d2851c","Type":"ContainerStarted","Data":"ed6c7a18a3d6454758827fb6eddadffda1c40c1028944e41c45e112eee327988"}
Nov 04 00:02:43.015008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:43.015000    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:43.051973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:43.051912    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" podStartSLOduration=2.05189648 podStartE2EDuration="2.05189648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:43.046856154 +0000 UTC m=+1978.937918090" watchObservedRunningTime="2026-11-04 00:02:43.05189648 +0000 UTC m=+1978.942958415"
Nov 04 00:02:44.469536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:44.469434    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:02:44.470067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:44.469727    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" podUID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" containerName="maas-api" containerID="cri-o://28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7" gracePeriod=30
Nov 04 00:02:44.978602 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:44.978572    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:02:45.036011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.035972    6430 generic.go:356] "Generic (PLEG): container finished" podID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" containerID="28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7" exitCode=0
Nov 04 00:02:45.036207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.036036    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"
Nov 04 00:02:45.036207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.036045    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" event={"ID":"fbaa8ccb-6200-4e69-a47a-2dc7035b6220","Type":"ContainerDied","Data":"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"}
Nov 04 00:02:45.036207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.036107    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6" event={"ID":"fbaa8ccb-6200-4e69-a47a-2dc7035b6220","Type":"ContainerDied","Data":"b5063d1c4ea4fd534cbb687c637b6ea777bf8c9370a004e0521a735a6757496c"}
Nov 04 00:02:45.036207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.036137    6430 scope.go:122] "RemoveContainer" containerID="28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"
Nov 04 00:02:45.048630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.048610    6430 scope.go:122] "RemoveContainer" containerID="28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"
Nov 04 00:02:45.048974 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:02:45.048950    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7\": container with ID starting with 28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7 not found: ID does not exist" containerID="28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"
Nov 04 00:02:45.049040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.048987    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7"} err="failed to get container status \"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7\": rpc error: code = NotFound desc = could not find container \"28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7\": container with ID starting with 28f07901b29da0d3aa2dfe80bf187afad3d9cf250d0b8bce3691856cd7af79e7 not found: ID does not exist"
Nov 04 00:02:45.077422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.077360    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") pod \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") "
Nov 04 00:02:45.077582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.077571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\" (UniqueName: \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\") pod \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") "
Nov 04 00:02:45.077731 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.077713    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\") pod \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") "
Nov 04 00:02:45.077942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.077869    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\") pod \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\" (UID: \"fbaa8ccb-6200-4e69-a47a-2dc7035b6220\") "
Nov 04 00:02:45.078077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.078054    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca" pod "fbaa8ccb-6200-4e69-a47a-2dc7035b6220" (UID: "fbaa8ccb-6200-4e69-a47a-2dc7035b6220"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:02:45.078535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.078515    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.079635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.079611    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls" pod "fbaa8ccb-6200-4e69-a47a-2dc7035b6220" (UID: "fbaa8ccb-6200-4e69-a47a-2dc7035b6220"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:02:45.079842 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.079826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr" pod "fbaa8ccb-6200-4e69-a47a-2dc7035b6220" (UID: "fbaa8ccb-6200-4e69-a47a-2dc7035b6220"). InnerVolumeSpecName "kube-api-access-xj9rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:02:45.079922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.079851    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs" pod "fbaa8ccb-6200-4e69-a47a-2dc7035b6220" (UID: "fbaa8ccb-6200-4e69-a47a-2dc7035b6220"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:02:45.190519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.181006    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9rr\" (UniqueName: \"kubernetes.io/projected/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-kube-api-access-xj9rr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.190519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.190320    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.190519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.190357    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa8ccb-6200-4e69-a47a-2dc7035b6220-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:02:45.368987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.368950    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:02:45.372708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:45.372683    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27f8a1-7459698b4-n9xm6"]
Nov 04 00:02:46.234706 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:46.234673    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" path="/var/lib/kubelet/pods/fbaa8ccb-6200-4e69-a47a-2dc7035b6220/volumes"
Nov 04 00:02:51.212567 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.212525    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:02:51.213696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.213677    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" containerName="maas-api"
Nov 04 00:02:51.213792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.213699    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" containerName="maas-api"
Nov 04 00:02:51.213867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.213855    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbaa8ccb-6200-4e69-a47a-2dc7035b6220" containerName="maas-api"
Nov 04 00:02:51.218156 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.218128    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.227450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.227370    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-86deca-openshift-default-dockercfg-dcfbl\""
Nov 04 00:02:51.232433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.231813    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.455437    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.455521    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.455626    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jz6\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.455773    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.455820    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.456076    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.456118    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.456144    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.459417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.456174    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.558644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.558601    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.558935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.558912    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559045    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559436    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559597    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559756 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jz6\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.559943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.559929    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.560118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.560099    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.562714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.562686    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.565847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.565822    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.569266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.566695    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.569616 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.567159    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.569758 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.567443    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.585910 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.585867    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.586591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.586559    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.587125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.587097    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.588359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.588336    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jz6\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\") pod \"e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") " pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:51.834335 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:51.834242    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:52.155823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:52.155787    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:02:52.156277 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:02:52.156253    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25d18f2_34e3_4912_8f9b_0cffcaec49da.slice/crio-09682d3ba6256b2b6ea4ec366ba7f4de2a94d053f1ba9ba090330a1b7260b163 WatchSource:0}: Error finding container 09682d3ba6256b2b6ea4ec366ba7f4de2a94d053f1ba9ba090330a1b7260b163: Status 404 returned error can't find the container with id 09682d3ba6256b2b6ea4ec366ba7f4de2a94d053f1ba9ba090330a1b7260b163
Nov 04 00:02:52.159182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:52.159147    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:52.159292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:52.159220    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:52.159292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:52.159270    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:53.112812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.112774    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:02:53.116801 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.116773    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.122162 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.122130    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-06ade4-openshift-default-dockercfg-tl79c\""
Nov 04 00:02:53.128637 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.128600    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerStarted","Data":"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"}
Nov 04 00:02:53.128813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.128788    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerStarted","Data":"09682d3ba6256b2b6ea4ec366ba7f4de2a94d053f1ba9ba090330a1b7260b163"}
Nov 04 00:02:53.134896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.134867    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:02:53.209019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.208966    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podStartSLOduration=2.208952313 podStartE2EDuration="2.208952313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:53.206735413 +0000 UTC m=+1989.097797360" watchObservedRunningTime="2026-11-04 00:02:53.208952313 +0000 UTC m=+1989.100014245"
Nov 04 00:02:53.212763 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.212728    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.212914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.212778    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zkx\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.212914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.212869    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.212914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.212909    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.213185 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.213161    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.213247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.213217    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.213357 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.213329    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.213526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.213369    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.213599 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.213534    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.314771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.314736    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.314964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.314800    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.314966    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315014    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315059    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315252 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315087    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315252 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315188    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315481    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315503    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315592    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.315741 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315644    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zkx\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.316063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315921    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.316063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.315946    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.316063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.316047    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.317867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.317821    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.317978 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.317955    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.362415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.359747    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zkx\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.362415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.360980    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\") pod \"e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") " pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.435467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.435103    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:53.835094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.835060    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:53.842189 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.842156    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:53.991221 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:02:53.991182    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba151b3b_b90e_4c46_a5d7_7b2464b0f9a6.slice/crio-7b18cdd52e08ce25931cc98898c00ecc24252e0aa52f4ea0a013f391cd3ff14a WatchSource:0}: Error finding container 7b18cdd52e08ce25931cc98898c00ecc24252e0aa52f4ea0a013f391cd3ff14a: Status 404 returned error can't find the container with id 7b18cdd52e08ce25931cc98898c00ecc24252e0aa52f4ea0a013f391cd3ff14a
Nov 04 00:02:53.991755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.991729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:02:53.994309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.994273    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:53.994433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.994344    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:53.994433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:53.994375    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:02:54.026516 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.026490    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:02:54.140920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.140865    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerStarted","Data":"db90c43dbea9eee24b8bd7d6f94d6bbd08092ac3b8c3351514487f58d2d8251c"}
Nov 04 00:02:54.141477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.140979    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerStarted","Data":"7b18cdd52e08ce25931cc98898c00ecc24252e0aa52f4ea0a013f391cd3ff14a"}
Nov 04 00:02:54.141875 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.141851    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:54.142526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.142504    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:02:54.178990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.178927    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podStartSLOduration=1.178907379 podStartE2EDuration="1.178907379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:54.168679855 +0000 UTC m=+1990.059741793" watchObservedRunningTime="2026-11-04 00:02:54.178907379 +0000 UTC m=+1990.069969317"
Nov 04 00:02:54.445461 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:54.442018    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:55.442172 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:55.442135    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:56.158332 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:56.158297    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:02:56.159305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:02:56.159281    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:03:04.834306 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:04.834264    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:03:07.608954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:07.608915    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:03:10.518113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:10.518062    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:03:12.610057 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:12.610008    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" podUID="c1adfc7a-5459-474a-83ae-25b93702a68f" containerName="payload-processing" containerID="cri-o://2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f" gracePeriod=25
Nov 04 00:03:12.975290 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:12.975259    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:03:13.111623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.111586    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\") pod \"c1adfc7a-5459-474a-83ae-25b93702a68f\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") "
Nov 04 00:03:13.111834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.111744    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\" (UniqueName: \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\") pod \"c1adfc7a-5459-474a-83ae-25b93702a68f\" (UID: \"c1adfc7a-5459-474a-83ae-25b93702a68f\") "
Nov 04 00:03:13.112055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.112028    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume" pod "c1adfc7a-5459-474a-83ae-25b93702a68f" (UID: "c1adfc7a-5459-474a-83ae-25b93702a68f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:03:13.112450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.112429    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfc7a-5459-474a-83ae-25b93702a68f-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:13.113912 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.113886    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs" pod "c1adfc7a-5459-474a-83ae-25b93702a68f" (UID: "c1adfc7a-5459-474a-83ae-25b93702a68f"). InnerVolumeSpecName "kube-api-access-x5hfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:03:13.214555 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.214518    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hfs\" (UniqueName: \"kubernetes.io/projected/c1adfc7a-5459-474a-83ae-25b93702a68f-kube-api-access-x5hfs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:13.385121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.385031    6430 generic.go:356] "Generic (PLEG): container finished" podID="c1adfc7a-5459-474a-83ae-25b93702a68f" containerID="2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f" exitCode=0
Nov 04 00:03:13.385121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.385102    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"
Nov 04 00:03:13.385375 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.385124    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" event={"ID":"c1adfc7a-5459-474a-83ae-25b93702a68f","Type":"ContainerDied","Data":"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"}
Nov 04 00:03:13.385375 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.385222    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl" event={"ID":"c1adfc7a-5459-474a-83ae-25b93702a68f","Type":"ContainerDied","Data":"67c0e11efdc1701fdf6deccd4be0ca8744660979822a1f66e4a67d6903ace3e9"}
Nov 04 00:03:13.385375 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.385253    6430 scope.go:122] "RemoveContainer" containerID="2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"
Nov 04 00:03:13.398326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.398304    6430 scope.go:122] "RemoveContainer" containerID="2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"
Nov 04 00:03:13.398700 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:13.398672    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f\": container with ID starting with 2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f not found: ID does not exist" containerID="2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"
Nov 04 00:03:13.398764 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.398714    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f"} err="failed to get container status \"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f\": rpc error: code = NotFound desc = could not find container \"2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f\": container with ID starting with 2bb2e10709dc4eac24d5f29ee168bfe56fbf74662c765f9dcf1c7de04ddd9e5f not found: ID does not exist"
Nov 04 00:03:13.413608 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.413577    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:03:13.423494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:13.423462    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18f5c0-6c94cbcc8d-67pcl"]
Nov 04 00:03:14.235547 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:14.235512    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1adfc7a-5459-474a-83ae-25b93702a68f" path="/var/lib/kubelet/pods/c1adfc7a-5459-474a-83ae-25b93702a68f/volumes"
Nov 04 00:03:33.876403 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.876339    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:03:33.877201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.877180    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c1adfc7a-5459-474a-83ae-25b93702a68f" containerName="payload-processing"
Nov 04 00:03:33.877201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.877199    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1adfc7a-5459-474a-83ae-25b93702a68f" containerName="payload-processing"
Nov 04 00:03:33.877371 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.877357    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1adfc7a-5459-474a-83ae-25b93702a68f" containerName="payload-processing"
Nov 04 00:03:33.883163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.883135    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:33.891526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.891495    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-06ade4-serving-cert\""
Nov 04 00:03:33.895037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.895009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:03:33.975045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.975004    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4h5g\" (UniqueName: \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:33.975230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.975052    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:33.975230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.975126    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:33.975314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:33.975274    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.078908 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.078864    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4h5g\" (UniqueName: \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.079219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.079192    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.079373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.079357    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.079713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.079687    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.081652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.080721    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.082477 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:34.082451    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-serving-cert: secret "maas-api-e2e-extmod-b-06ade4-serving-cert" not found
Nov 04 00:03:34.082979 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:34.082770    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls podName:871e77b8-0d34-4fdc-aeac-1b110a9a1eba nodeName:}" failed. No retries permitted until 2026-11-04 00:03:34.582739499 +0000 UTC m=+2030.473801415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls") pod "maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" (UID: "871e77b8-0d34-4fdc-aeac-1b110a9a1eba") : secret "maas-api-e2e-extmod-b-06ade4-serving-cert" not found
Nov 04 00:03:34.127536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.116899    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4h5g\" (UniqueName: \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.128043 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.128005    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.589192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.589119    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.592085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.592059    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:34.828140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:34.828103    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:35.050119 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:35.050077    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:03:35.405433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:35.388638    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:03:35.643317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:35.643199    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" event={"ID":"871e77b8-0d34-4fdc-aeac-1b110a9a1eba","Type":"ContainerStarted","Data":"322f85f2aef45ccaff3f324aebd3f4b300f87c6aafda01a4476ddd7ab61de90a"}
Nov 04 00:03:36.652247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:36.652204    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" event={"ID":"871e77b8-0d34-4fdc-aeac-1b110a9a1eba","Type":"ContainerStarted","Data":"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"}
Nov 04 00:03:36.652702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:36.652312    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:36.692954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:36.692894    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" podStartSLOduration=3.220974107 podStartE2EDuration="3.692877816s" totalImagesPullingTime="471.903709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:36.689522049 +0000 UTC m=+2032.580583984" watchObservedRunningTime="2026-11-04 00:03:36.692877816 +0000 UTC m=+2032.583939750"
Nov 04 00:03:36.697266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:36.697236    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:03:36.697482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:36.697456    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" podUID="fa8ec593-5a98-47cb-addd-eaf90029098d" containerName="maas-api" containerID="cri-o://5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0" gracePeriod=30
Nov 04 00:03:37.217243 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.216881    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:03:37.377630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.373275    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\") pod \"fa8ec593-5a98-47cb-addd-eaf90029098d\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") "
Nov 04 00:03:37.377630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.373448    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\") pod \"fa8ec593-5a98-47cb-addd-eaf90029098d\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") "
Nov 04 00:03:37.377630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.373540    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\" (UniqueName: \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\") pod \"fa8ec593-5a98-47cb-addd-eaf90029098d\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") "
Nov 04 00:03:37.377630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.373582    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\") pod \"fa8ec593-5a98-47cb-addd-eaf90029098d\" (UID: \"fa8ec593-5a98-47cb-addd-eaf90029098d\") "
Nov 04 00:03:37.377630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.377250    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca" pod "fa8ec593-5a98-47cb-addd-eaf90029098d" (UID: "fa8ec593-5a98-47cb-addd-eaf90029098d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:03:37.379561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.379525    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9" pod "fa8ec593-5a98-47cb-addd-eaf90029098d" (UID: "fa8ec593-5a98-47cb-addd-eaf90029098d"). InnerVolumeSpecName "kube-api-access-bgdh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:03:37.379970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.379939    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls" pod "fa8ec593-5a98-47cb-addd-eaf90029098d" (UID: "fa8ec593-5a98-47cb-addd-eaf90029098d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:03:37.380620 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.380597    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs" pod "fa8ec593-5a98-47cb-addd-eaf90029098d" (UID: "fa8ec593-5a98-47cb-addd-eaf90029098d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:03:37.478134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.478035    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:37.478134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.478075    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgdh9\" (UniqueName: \"kubernetes.io/projected/fa8ec593-5a98-47cb-addd-eaf90029098d-kube-api-access-bgdh9\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:37.478134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.478093    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8ec593-5a98-47cb-addd-eaf90029098d-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:37.478134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.478109    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa8ec593-5a98-47cb-addd-eaf90029098d-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:03:37.676363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.675988    6430 generic.go:356] "Generic (PLEG): container finished" podID="fa8ec593-5a98-47cb-addd-eaf90029098d" containerID="5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0" exitCode=0
Nov 04 00:03:37.677002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.676978    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"
Nov 04 00:03:37.677266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.677139    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" event={"ID":"fa8ec593-5a98-47cb-addd-eaf90029098d","Type":"ContainerDied","Data":"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"}
Nov 04 00:03:37.677348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.677308    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf" event={"ID":"fa8ec593-5a98-47cb-addd-eaf90029098d","Type":"ContainerDied","Data":"8bfcbdc0f2a91e47d13babb299a7fcbb8b391b08377be17087460c863e0b3d60"}
Nov 04 00:03:37.677348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.677336    6430 scope.go:122] "RemoveContainer" containerID="5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"
Nov 04 00:03:37.689459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.689436    6430 scope.go:122] "RemoveContainer" containerID="5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"
Nov 04 00:03:37.689735 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:03:37.689712    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0\": container with ID starting with 5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0 not found: ID does not exist" containerID="5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"
Nov 04 00:03:37.689836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.689742    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0"} err="failed to get container status \"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0\": rpc error: code = NotFound desc = could not find container \"5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0\": container with ID starting with 5b0d7f610cd2ed54d4fd1d00f7260ff75e118e54467e4e55a4d077488e28f7b0 not found: ID does not exist"
Nov 04 00:03:37.704810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.704775    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:03:37.711423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:37.711373    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18f5c0-59f5cdccdf-vkgxf"]
Nov 04 00:03:38.235017 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:38.234977    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8ec593-5a98-47cb-addd-eaf90029098d" path="/var/lib/kubelet/pods/fa8ec593-5a98-47cb-addd-eaf90029098d/volumes"
Nov 04 00:03:42.683168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:42.683120    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:03:47.500417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.486361    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:03:47.500417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.487686    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fa8ec593-5a98-47cb-addd-eaf90029098d" containerName="maas-api"
Nov 04 00:03:47.500417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.487713    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8ec593-5a98-47cb-addd-eaf90029098d" containerName="maas-api"
Nov 04 00:03:47.500417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.487939    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8ec593-5a98-47cb-addd-eaf90029098d" containerName="maas-api"
Nov 04 00:03:47.500417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.494100    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.512448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.510601    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-71b2f3-openshift-default-dockercfg-nhn7l\""
Nov 04 00:03:47.517417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.513913    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670408    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670532    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670581    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pbh\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670611    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670681    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670734    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670816    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670864    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.691451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.670901    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.789477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.772777    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.789859 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.789823    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.790048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.790028    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pbh\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.790194 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.790172    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.790411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.790376    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.793507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.793482    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.793750 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.793732    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.793888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.793874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.794014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.793999    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.802320    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.802994    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.803301    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.821657    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.824835    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.826406    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.832486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.830531    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pbh\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.842739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.842660    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:47.843234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:47.843198    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\") pod \"e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") " pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:48.139069 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.139036    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:48.432937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.432900    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:03:48.434249 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:03:48.434220    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4111f9a_5247_4406_889f_e5b99a2e9720.slice/crio-f2a36ef1513af47f94cf211da889e99e39cdbf49a7fd1ee34ad6c937aa2a3e1d WatchSource:0}: Error finding container f2a36ef1513af47f94cf211da889e99e39cdbf49a7fd1ee34ad6c937aa2a3e1d: Status 404 returned error can't find the container with id f2a36ef1513af47f94cf211da889e99e39cdbf49a7fd1ee34ad6c937aa2a3e1d
Nov 04 00:03:48.437359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.437323    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:03:48.437481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.437440    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:03:48.437481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.437475    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:03:48.852914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.852861    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerStarted","Data":"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"}
Nov 04 00:03:48.853366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.852934    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerStarted","Data":"f2a36ef1513af47f94cf211da889e99e39cdbf49a7fd1ee34ad6c937aa2a3e1d"}
Nov 04 00:03:48.883213 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:48.883152    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podStartSLOduration=1.883133253 podStartE2EDuration="1.883133253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:48.879491566 +0000 UTC m=+2044.770553504" watchObservedRunningTime="2026-11-04 00:03:48.883133253 +0000 UTC m=+2044.774195190"
Nov 04 00:03:49.140283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:49.140187    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:49.147077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:49.147047    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:49.862067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:49.862034    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:49.863120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:49.863097    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:03:58.704989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:03:58.704948    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:04:04.905510 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:04:04.905466    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:04:10.183630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:10.183579    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:04:10.184219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:10.183678    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:04:11.160138 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.160092    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:11.160383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.160163    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:11.407137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.407104    6430 generic.go:356] "Generic (PLEG): container finished" podID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerID="db90c43dbea9eee24b8bd7d6f94d6bbd08092ac3b8c3351514487f58d2d8251c" exitCode=0
Nov 04 00:04:11.407628 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.407180    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerDied","Data":"db90c43dbea9eee24b8bd7d6f94d6bbd08092ac3b8c3351514487f58d2d8251c"}
Nov 04 00:04:11.407765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.407746    6430 scope.go:122] "RemoveContainer" containerID="db90c43dbea9eee24b8bd7d6f94d6bbd08092ac3b8c3351514487f58d2d8251c"
Nov 04 00:04:11.409098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.409080    6430 generic.go:356] "Generic (PLEG): container finished" podID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerID="e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd" exitCode=0
Nov 04 00:04:11.409188 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.409150    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerDied","Data":"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"}
Nov 04 00:04:11.409639 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.409621    6430 scope.go:122] "RemoveContainer" containerID="e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"
Nov 04 00:04:11.409992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.409962    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:11.410074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.410054    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:11.410128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.410114    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:11.411527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.411499    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:11.411663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.411566    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:11.411663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:11.411599    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:04:12.463035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:12.462986    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerStarted","Data":"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"}
Nov 04 00:04:12.469193 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:12.469145    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerStarted","Data":"d550941ecc510536446683a3dc21b3397dff925aa90f8fa1471fc9c752b9cbf5"}
Nov 04 00:04:12.849408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:12.847412    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:04:12.849408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:12.848557    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:12.849408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:12.848612    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:13.440413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:13.439882    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:04:13.440413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:13.440028    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:13.440413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:13.440088    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:13.846412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:13.845068    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:13.846412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:13.845132    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:14.443420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:14.438888    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:14.443420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:14.438960    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:14.857411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:14.853265    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:14.857411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:14.853332    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:15.438413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:15.436461    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:15.438413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:15.436545    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:15.836899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:15.836849    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:15.837717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:15.837219    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:16.440419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:16.439949    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:16.440419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:16.440036    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:16.839829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:16.839786    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:16.840153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:16.840122    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:17.436467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:17.436351    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:17.436809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:17.436775    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:17.838716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:17.838671    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 04 00:04:17.839427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:17.839366    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 04 00:04:18.438852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:18.438815    6430 patch_prober.go:29] interesting pod/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 04 00:04:18.440577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:18.440538    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 04 00:04:19.063120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.063085    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:04:19.064501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.064460    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:04:19.066054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.066029    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:04:19.555479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.525020    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:04:19.555479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.534846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:04:19.595337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:19.574556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:04:32.991891 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:32.991848    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:04:32.998254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:32.998223    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.010090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.010050    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:04:33.010326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.010303    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-71b2f3-serving-cert\""
Nov 04 00:04:33.184440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.183640    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.184440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.183715    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.184440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.184229    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.184824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.184596    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2qq\" (UniqueName: \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.285973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.285874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.286136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.286045    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2qq\" (UniqueName: \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.286136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.286101    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.286227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.286136    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.286611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.286587    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.288909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.288884    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.289027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.288884    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.299444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.299417    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2qq\" (UniqueName: \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\") pod \"maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.345286 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.345251    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:33.661504 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.661454    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:04:33.662333 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:04:33.662307    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a79208_d133_4c26_90dd_364f5146ee8a.slice/crio-2bba8d5b767c03ceb949d0cdcc5d357bdd8d5e7163a881c0c37be0ce4ec1be91 WatchSource:0}: Error finding container 2bba8d5b767c03ceb949d0cdcc5d357bdd8d5e7163a881c0c37be0ce4ec1be91: Status 404 returned error can't find the container with id 2bba8d5b767c03ceb949d0cdcc5d357bdd8d5e7163a881c0c37be0ce4ec1be91
Nov 04 00:04:33.882108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:33.882065    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerStarted","Data":"2bba8d5b767c03ceb949d0cdcc5d357bdd8d5e7163a881c0c37be0ce4ec1be91"}
Nov 04 00:04:34.833519 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:04:34.833479    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:04:34.891965 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:34.891933    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerStarted","Data":"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"}
Nov 04 00:04:34.892146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:34.892071    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:34.914637 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:34.914582    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" podStartSLOduration=2.453400476 podStartE2EDuration="2.91456916s" totalImagesPullingTime="461.168684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:34.913366495 +0000 UTC m=+2090.804428431" watchObservedRunningTime="2026-11-04 00:04:34.91456916 +0000 UTC m=+2090.805631096"
Nov 04 00:04:37.357494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:37.357456    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:04:40.901040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:40.900990    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.186:8443/health\": dial tcp 10.134.0.186:8443: connect: connection refused"
Nov 04 00:04:41.502752 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:41.502710    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" podUID="b6a44b6090d9fc279a15fb8a9a158ae7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 04 00:04:43.330284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:43.330224    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:04:45.026643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:45.026605    6430 generic.go:356] "Generic (PLEG): container finished" podID="61a79208-d133-4c26-90dd-364f5146ee8a" containerID="9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde" exitCode=1
Nov 04 00:04:45.027079 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:45.026681    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerDied","Data":"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"}
Nov 04 00:04:45.027241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:45.027225    6430 scope.go:122] "RemoveContainer" containerID="9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"
Nov 04 00:04:46.036173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:46.036133    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerStarted","Data":"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"}
Nov 04 00:04:46.036635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:46.036353    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:04:48.331326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.331286    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" podUID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" containerName="payload-processing" containerID="cri-o://031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f" gracePeriod=25
Nov 04 00:04:48.633564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.633535    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:04:48.731346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.731291    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\") pod \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") "
Nov 04 00:04:48.731635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.731460    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\") pod \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") "
Nov 04 00:04:48.731635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.731490    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\") pod \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") "
Nov 04 00:04:48.731635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.731574    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\") pod \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\" (UID: \"9be52f8e-593b-43d3-9f1e-9fe715d2851c\") "
Nov 04 00:04:48.731953 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.731921    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume" pod "9be52f8e-593b-43d3-9f1e-9fe715d2851c" (UID: "9be52f8e-593b-43d3-9f1e-9fe715d2851c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:04:48.732033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.732006    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume" pod "9be52f8e-593b-43d3-9f1e-9fe715d2851c" (UID: "9be52f8e-593b-43d3-9f1e-9fe715d2851c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:04:48.732378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.732361    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:04:48.732446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.732382    6430 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9be52f8e-593b-43d3-9f1e-9fe715d2851c-routing-overlay-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:04:48.733702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.733680    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd" pod "9be52f8e-593b-43d3-9f1e-9fe715d2851c" (UID: "9be52f8e-593b-43d3-9f1e-9fe715d2851c"). InnerVolumeSpecName "kube-api-access-bkwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:04:48.733798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.733766    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials" pod "9be52f8e-593b-43d3-9f1e-9fe715d2851c" (UID: "9be52f8e-593b-43d3-9f1e-9fe715d2851c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:04:48.834156 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.834113    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwhd\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-kube-api-access-bkwhd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:04:48.834156 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:48.834145    6430 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9be52f8e-593b-43d3-9f1e-9fe715d2851c-provider-credentials\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:04:49.062564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.062528    6430 generic.go:356] "Generic (PLEG): container finished" podID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" containerID="031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f" exitCode=0
Nov 04 00:04:49.062877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.062583    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"
Nov 04 00:04:49.062877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.062591    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" event={"ID":"9be52f8e-593b-43d3-9f1e-9fe715d2851c","Type":"ContainerDied","Data":"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"}
Nov 04 00:04:49.062877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.062649    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t" event={"ID":"9be52f8e-593b-43d3-9f1e-9fe715d2851c","Type":"ContainerDied","Data":"ed6c7a18a3d6454758827fb6eddadffda1c40c1028944e41c45e112eee327988"}
Nov 04 00:04:49.062877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.062675    6430 scope.go:122] "RemoveContainer" containerID="031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"
Nov 04 00:04:49.086014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.085984    6430 scope.go:122] "RemoveContainer" containerID="031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"
Nov 04 00:04:49.086381 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:04:49.086356    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f\": container with ID starting with 031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f not found: ID does not exist" containerID="031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"
Nov 04 00:04:49.086468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.086406    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f"} err="failed to get container status \"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f\": rpc error: code = NotFound desc = could not find container \"031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f\": container with ID starting with 031e9846a47107a2592634064f078ec4928b43c95f5c81c05e1c64915181cc6f not found: ID does not exist"
Nov 04 00:04:49.098968 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.098937    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:04:49.105155 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:49.105121    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-86c77sl5t"]
Nov 04 00:04:50.278541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:50.278500    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" path="/var/lib/kubelet/pods/9be52f8e-593b-43d3-9f1e-9fe715d2851c/volumes"
Nov 04 00:04:52.050709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:04:52.050679    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:05:04.301055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.301021    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:04.301843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.301827    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" containerName="payload-processing"
Nov 04 00:05:04.301904 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.301844    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" containerName="payload-processing"
Nov 04 00:05:04.301992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.301983    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be52f8e-593b-43d3-9f1e-9fe715d2851c" containerName="payload-processing"
Nov 04 00:05:04.305880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.305858    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.314857 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.314823    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-58159406142a\""
Nov 04 00:05:04.314857 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.314826    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-acf8cd35b468\""
Nov 04 00:05:04.314857 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.314826    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-e2e--5d63ff17-dockercfg-lrbbg\""
Nov 04 00:05:04.317303 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.317267    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:04.317456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.317435    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"kube-root-ca.crt\""
Nov 04 00:05:04.317530 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.317514    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2697a0\""
Nov 04 00:05:04.317672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.317565    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"openshift-service-ca.crt\""
Nov 04 00:05:04.317672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.317576    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"routing-overlay\""
Nov 04 00:05:04.437976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.437940    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.438173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.438056    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf89\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.438230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.438213    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.438319 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.438307    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.539467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.539422    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.539683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.539502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf89\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.539683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.539608    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.539833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.539690    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.540344 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.540314    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.540420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.540349    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.542280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.542259    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.553492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.553409    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf89\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.636412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:04.629253    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:04.851118 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:05:04.851005    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:05:05.011701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:05.011668    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:05.013177 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:05:05.013147    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0332c_7e71_445f_a897_09a5b8989593.slice/crio-3691489b29f0e39af9ea89ae9c8a648c43113531c6774fc6cc990494c569ddde WatchSource:0}: Error finding container 3691489b29f0e39af9ea89ae9c8a648c43113531c6774fc6cc990494c569ddde: Status 404 returned error can't find the container with id 3691489b29f0e39af9ea89ae9c8a648c43113531c6774fc6cc990494c569ddde
Nov 04 00:05:05.203241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:05.202590    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" event={"ID":"62c0332c-7e71-445f-a897-09a5b8989593","Type":"ContainerStarted","Data":"36b15f63a43681378061bb8d26e64f9028fe4c130e6a399d97cb0dd41ef0d21a"}
Nov 04 00:05:05.203241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:05.202664    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" event={"ID":"62c0332c-7e71-445f-a897-09a5b8989593","Type":"ContainerStarted","Data":"3691489b29f0e39af9ea89ae9c8a648c43113531c6774fc6cc990494c569ddde"}
Nov 04 00:05:05.240409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:05.215915    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:06.027683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:06.027623    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" podStartSLOduration=2.027607064 podStartE2EDuration="2.027607064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:05.261103817 +0000 UTC m=+2121.152165753" watchObservedRunningTime="2026-11-04 00:05:06.027607064 +0000 UTC m=+2121.918669044"
Nov 04 00:05:06.031102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:06.031068    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"]
Nov 04 00:05:06.035113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:06.035090    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-783d53cd-tqvpg"]
Nov 04 00:05:06.233894 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:06.233858    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1c8fe0-5cd7-418d-8104-286f072e8f34" path="/var/lib/kubelet/pods/3b1c8fe0-5cd7-418d-8104-286f072e8f34/volumes"
Nov 04 00:05:16.036694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:16.036650    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"]
Nov 04 00:05:16.045255 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:16.045215    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d90d111-cbgfg"]
Nov 04 00:05:16.236921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:16.236832    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ea8423-e44d-438e-9c5d-ede8931cfffa" path="/var/lib/kubelet/pods/59ea8423-e44d-438e-9c5d-ede8931cfffa/volumes"
Nov 04 00:05:17.236970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:17.236925    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:19.924249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:19.922787    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": read tcp 10.134.0.2:51048->10.134.0.185:15021: read: connection reset by peer" start-of-body=
Nov 04 00:05:19.924249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:19.922851    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": read tcp 10.134.0.2:51048->10.134.0.185:15021: read: connection reset by peer"
Nov 04 00:05:20.486772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.486735    6430 generic.go:356] "Generic (PLEG): container finished" podID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerID="0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a" exitCode=0
Nov 04 00:05:20.487061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.486815    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerDied","Data":"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"}
Nov 04 00:05:20.487379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.487353    6430 scope.go:122] "RemoveContainer" containerID="0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"
Nov 04 00:05:20.489568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.489539    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:05:20.489641 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.489614    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:05:20.489679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:20.489649    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:05:21.496971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:21.496933    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerStarted","Data":"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"}
Nov 04 00:05:22.139874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:22.139829    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:05:22.140154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:22.140127    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:22.140228 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:22.140203    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:23.141414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:23.140827    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:23.141414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:23.140906    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:24.146823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:24.146781    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:24.147507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:24.147471    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:25.139951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:25.139912    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:25.140119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:25.140001    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:26.140093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:26.140046    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:26.140531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:26.140126    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:27.140338 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:27.140297    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 04 00:05:27.140886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:27.140366    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 04 00:05:28.143793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:28.143760    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:05:28.144364 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:28.144140    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:05:28.144825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:28.144803    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:05:34.234405 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:34.234353    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:05:34.836144 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:05:34.836097    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:05:36.972836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:36.972794    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:41.973287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:41.973217    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" podUID="62c0332c-7e71-445f-a897-09a5b8989593" containerName="payload-processing" containerID="cri-o://36b15f63a43681378061bb8d26e64f9028fe4c130e6a399d97cb0dd41ef0d21a" gracePeriod=25
Nov 04 00:05:42.116762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.116730    6430 generic.go:356] "Generic (PLEG): container finished" podID="62c0332c-7e71-445f-a897-09a5b8989593" containerID="36b15f63a43681378061bb8d26e64f9028fe4c130e6a399d97cb0dd41ef0d21a" exitCode=0
Nov 04 00:05:42.116959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.116803    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" event={"ID":"62c0332c-7e71-445f-a897-09a5b8989593","Type":"ContainerDied","Data":"36b15f63a43681378061bb8d26e64f9028fe4c130e6a399d97cb0dd41ef0d21a"}
Nov 04 00:05:42.273103 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.273071    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:42.374594 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.374549    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\") pod \"62c0332c-7e71-445f-a897-09a5b8989593\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") "
Nov 04 00:05:42.374802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.374671    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\") pod \"62c0332c-7e71-445f-a897-09a5b8989593\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") "
Nov 04 00:05:42.374864 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.374808    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\") pod \"62c0332c-7e71-445f-a897-09a5b8989593\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") "
Nov 04 00:05:42.374911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.374861    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\") pod \"62c0332c-7e71-445f-a897-09a5b8989593\" (UID: \"62c0332c-7e71-445f-a897-09a5b8989593\") "
Nov 04 00:05:42.375067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.375038    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume" pod "62c0332c-7e71-445f-a897-09a5b8989593" (UID: "62c0332c-7e71-445f-a897-09a5b8989593"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:42.375407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.375365    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume" pod "62c0332c-7e71-445f-a897-09a5b8989593" (UID: "62c0332c-7e71-445f-a897-09a5b8989593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:05:42.375701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.375680    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:42.375767 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.375707    6430 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c0332c-7e71-445f-a897-09a5b8989593-routing-overlay-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:42.377215 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.377188    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials" pod "62c0332c-7e71-445f-a897-09a5b8989593" (UID: "62c0332c-7e71-445f-a897-09a5b8989593"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:42.377358 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.377333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89" pod "62c0332c-7e71-445f-a897-09a5b8989593" (UID: "62c0332c-7e71-445f-a897-09a5b8989593"). InnerVolumeSpecName "kube-api-access-drf89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:05:42.478022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.477986    6430 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-provider-credentials\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:42.478022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:42.478018    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drf89\" (UniqueName: \"kubernetes.io/projected/62c0332c-7e71-445f-a897-09a5b8989593-kube-api-access-drf89\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:05:43.126147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:43.126114    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"
Nov 04 00:05:43.126147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:43.126133    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q" event={"ID":"62c0332c-7e71-445f-a897-09a5b8989593","Type":"ContainerDied","Data":"3691489b29f0e39af9ea89ae9c8a648c43113531c6774fc6cc990494c569ddde"}
Nov 04 00:05:43.126730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:43.126197    6430 scope.go:122] "RemoveContainer" containerID="36b15f63a43681378061bb8d26e64f9028fe4c130e6a399d97cb0dd41ef0d21a"
Nov 04 00:05:43.156921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:43.156888    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:43.160630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:43.160604    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d98hr46q"]
Nov 04 00:05:44.234047 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:44.234015    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c0332c-7e71-445f-a897-09a5b8989593" path="/var/lib/kubelet/pods/62c0332c-7e71-445f-a897-09a5b8989593/volumes"
Nov 04 00:05:59.125064 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.125029    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:05:59.125858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.125841    6430 cpu_manager.go:414] "containerMap: removing container" podUID="62c0332c-7e71-445f-a897-09a5b8989593" containerName="payload-processing"
Nov 04 00:05:59.125915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.125859    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0332c-7e71-445f-a897-09a5b8989593" containerName="payload-processing"
Nov 04 00:05:59.126011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.126002    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0332c-7e71-445f-a897-09a5b8989593" containerName="payload-processing"
Nov 04 00:05:59.129710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.129690    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.138551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.138508    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"openshift-service-ca.crt\""
Nov 04 00:05:59.140088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.140018    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4376d\""
Nov 04 00:05:59.140088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.140042    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"payload-processing-external-model-e2e--db373d70-dockercfg-s84x2\""
Nov 04 00:05:59.140088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.140081    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"routing-overlay\""
Nov 04 00:05:59.142094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.140120    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"agc-cred-76b8ab897db6\""
Nov 04 00:05:59.142094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.140150    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4376d\"/\"kube-root-ca.crt\""
Nov 04 00:05:59.147447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.147409    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:05:59.234954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.234912    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.235151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.235072    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.235151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.235130    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.235345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.235330    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpzv\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.345447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.337574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.353410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.345967    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpzv\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.357409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.353746    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.361410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.357799    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.369412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.367422    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.369412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.341536    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.369412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.368321    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.406853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.406738    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpzv\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\") pod \"payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") " pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.454410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.450587    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:05:59.859145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:05:59.859111    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:05:59.859902 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:05:59.859874    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf94a599_d8ba_42df_a9fc_c0664c5c80ef.slice/crio-22aaa6787005a632d9adc84d893f742efb1669dd8d0a375f94458f0fae548a5d WatchSource:0}: Error finding container 22aaa6787005a632d9adc84d893f742efb1669dd8d0a375f94458f0fae548a5d: Status 404 returned error can't find the container with id 22aaa6787005a632d9adc84d893f742efb1669dd8d0a375f94458f0fae548a5d
Nov 04 00:06:00.356462 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:00.356424    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" event={"ID":"af94a599-d8ba-42df-a9fc-c0664c5c80ef","Type":"ContainerStarted","Data":"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"}
Nov 04 00:06:00.356862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:00.356488    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" event={"ID":"af94a599-d8ba-42df-a9fc-c0664c5c80ef","Type":"ContainerStarted","Data":"22aaa6787005a632d9adc84d893f742efb1669dd8d0a375f94458f0fae548a5d"}
Nov 04 00:06:00.356862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:00.356626    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:06:00.378642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:00.378584    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" podStartSLOduration=1.378569738 podStartE2EDuration="1.378569738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:06:00.377709058 +0000 UTC m=+2176.268771016" watchObservedRunningTime="2026-11-04 00:06:00.378569738 +0000 UTC m=+2176.269631664"
Nov 04 00:06:04.833483 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:06:04.833437    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:06:11.366207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:11.366174    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:06:18.903173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:18.903034    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 04 00:06:18.904181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:18.904111    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy" containerID="cri-o://12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16" gracePeriod=30
Nov 04 00:06:24.269121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.269054    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 04 00:06:24.449004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.448968    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449068    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449101    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449257    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449278    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449303    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:24.449404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449322    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449422    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449442    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449468    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449556    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:24.449656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449618    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\") pod \"8abf9512-7c01-4031-a73f-f6fae8939179\" (UID: \"8abf9512-7c01-4031-a73f-f6fae8939179\") "
Nov 04 00:06:24.449999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.449719    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:24.450581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.450554    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.450709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.450583    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8abf9512-7c01-4031-a73f-f6fae8939179-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.450709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.450602    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.450709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.450617    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.450709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.450633    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.451837 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.451801    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:24.451962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.451925    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "kube-api-access-g9qnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:24.452042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.452023    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:24.452106 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.452089    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo" pod "8abf9512-7c01-4031-a73f-f6fae8939179" (UID: "8abf9512-7c01-4031-a73f-f6fae8939179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:24.552025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.551993    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.552206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.552029    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8abf9512-7c01-4031-a73f-f6fae8939179-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.552206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.552042    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8abf9512-7c01-4031-a73f-f6fae8939179-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.552206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.552052    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9qnt\" (UniqueName: \"kubernetes.io/projected/8abf9512-7c01-4031-a73f-f6fae8939179-kube-api-access-g9qnt\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:24.650518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.650482    6430 generic.go:356] "Generic (PLEG): container finished" podID="8abf9512-7c01-4031-a73f-f6fae8939179" containerID="12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16" exitCode=0
Nov 04 00:06:24.650697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.650533    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"
Nov 04 00:06:24.650697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.650574    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" event={"ID":"8abf9512-7c01-4031-a73f-f6fae8939179","Type":"ContainerDied","Data":"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"}
Nov 04 00:06:24.650697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.650637    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph" event={"ID":"8abf9512-7c01-4031-a73f-f6fae8939179","Type":"ContainerDied","Data":"963830819f023bdbef6bdd108fb6f2791ae788231a5e29f820272ad69d7ba526"}
Nov 04 00:06:24.650697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.650665    6430 scope.go:122] "RemoveContainer" containerID="12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"
Nov 04 00:06:24.662743 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.662725    6430 scope.go:122] "RemoveContainer" containerID="12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"
Nov 04 00:06:24.663005 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:06:24.662988    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16\": container with ID starting with 12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16 not found: ID does not exist" containerID="12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"
Nov 04 00:06:24.663065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.663013    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16"} err="failed to get container status \"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16\": rpc error: code = NotFound desc = could not find container \"12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16\": container with ID starting with 12a15ad0b6288ba457c8f457cbb04f58385b119b9a64300cf0cd0569440dbc16 not found: ID does not exist"
Nov 04 00:06:24.678729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.678677    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 04 00:06:24.683471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:24.683443    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-gspph"]
Nov 04 00:06:26.238184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:26.238145    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" path="/var/lib/kubelet/pods/8abf9512-7c01-4031-a73f-f6fae8939179/volumes"
Nov 04 00:06:34.835088 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:06:34.835046    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:06:44.052018 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:44.051981    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:06:53.741214 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:53.741180    6430 scope.go:122] "RemoveContainer" containerID="1a83b45d21f877ac27b82a22dde6a6c7f2cee376cbdd1a5301314de73537fcbf"
Nov 04 00:06:53.752339 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:53.752317    6430 scope.go:122] "RemoveContainer" containerID="8366eefc9a599fff14205559af8f70990d15c66554d3ab9a8be4d58149f4f7c1"
Nov 04 00:06:53.767029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:53.767003    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 04 00:06:53.767236 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:53.767212    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy" containerID="cri-o://149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16" gracePeriod=30
Nov 04 00:06:59.129920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.129885    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 04 00:06:59.156572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.156537    6430 generic.go:356] "Generic (PLEG): container finished" podID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerID="149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16" exitCode=0
Nov 04 00:06:59.156755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.156582    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"
Nov 04 00:06:59.156755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.156614    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" event={"ID":"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d","Type":"ContainerDied","Data":"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"}
Nov 04 00:06:59.156755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.156666    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq" event={"ID":"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d","Type":"ContainerDied","Data":"a38a96041cb72de832a87b97a9863bac8cf43c415cfaff05f80e8c5c3da40d37"}
Nov 04 00:06:59.156755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.156689    6430 scope.go:122] "RemoveContainer" containerID="149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"
Nov 04 00:06:59.170414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.170380    6430 scope.go:122] "RemoveContainer" containerID="149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"
Nov 04 00:06:59.170752 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:06:59.170727    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16\": container with ID starting with 149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16 not found: ID does not exist" containerID="149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"
Nov 04 00:06:59.170836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.170763    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16"} err="failed to get container status \"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16\": rpc error: code = NotFound desc = could not find container \"149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16\": container with ID starting with 149e294e64e41eac49a478ab53a6826f745ca1203edb9865be4b019a52c80d16 not found: ID does not exist"
Nov 04 00:06:59.257097 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257064    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257282 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257124    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257282 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257205    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257307    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257343    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257402    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:59.257484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257450    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257571 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257496    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257571 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257447    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:59.257571 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257549    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257585    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\") pod \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\" (UID: \"88a1d1ca-ecec-41a0-9642-e41e84f9dd1d\") "
Nov 04 00:06:59.257810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257783    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:59.257894 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257798    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:59.257996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.257964    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:06:59.258380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.258363    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.258380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.258381    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.258557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.258412    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.258557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.258424    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.258557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.258434    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.259837 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.259815    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "kube-api-access-5qpcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:59.259918 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.259842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:06:59.259997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.259977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:06:59.260039 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.259993    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo" pod "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" (UID: "88a1d1ca-ecec-41a0-9642-e41e84f9dd1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:06:59.360196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.360100    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.360196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.360135    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.360196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.360145    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.360196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.360154    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpcq\" (UniqueName: \"kubernetes.io/projected/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d-kube-api-access-5qpcq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:06:59.488377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.488331    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 04 00:06:59.492085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:06:59.492053    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-gc6nq"]
Nov 04 00:07:00.233626 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:00.233590    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" path="/var/lib/kubelet/pods/88a1d1ca-ecec-41a0-9642-e41e84f9dd1d/volumes"
Nov 04 00:07:02.078181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:02.078144    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 04 00:07:02.078694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:02.078356    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" containerID="cri-o://23a95723a45269ae4293c85798db374f1383aa98b03a7c5fd37fe9564a64349f" gracePeriod=30
Nov 04 00:07:04.833901 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:07:04.833862    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:07:06.436542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:06.436503    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:07:06.437129 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:06.437095    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:07:07.298204 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.298169    6430 generic.go:356] "Generic (PLEG): container finished" podID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerID="23a95723a45269ae4293c85798db374f1383aa98b03a7c5fd37fe9564a64349f" exitCode=0
Nov 04 00:07:07.298443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.298242    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" event={"ID":"7286d5c3-0565-45a2-bc50-20d67023ce69","Type":"ContainerDied","Data":"23a95723a45269ae4293c85798db374f1383aa98b03a7c5fd37fe9564a64349f"}
Nov 04 00:07:07.766796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.766767    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 04 00:07:07.865593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865488    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.865593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865564    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.865838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865626    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.865838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865686    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.865838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865711    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.865838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865757    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.866070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865879    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.866070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865938    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:07.866070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.865977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:07.866226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.866097    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:07.866226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.866159    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:07:07.866336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.866225    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.866336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.866271    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\") pod \"7286d5c3-0565-45a2-bc50-20d67023ce69\" (UID: \"7286d5c3-0565-45a2-bc50-20d67023ce69\") "
Nov 04 00:07:07.866703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.866676    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:07.867183 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.867157    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.867289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.867188    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.867289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.867205    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7286d5c3-0565-45a2-bc50-20d67023ce69-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.867289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.867218    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.867289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.867232    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.868211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.868182    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "kube-api-access-l52fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:07.868316 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.868259    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:07:07.868438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.868349    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:07:07.868656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.868637    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo" pod "7286d5c3-0565-45a2-bc50-20d67023ce69" (UID: "7286d5c3-0565-45a2-bc50-20d67023ce69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:07:07.968991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.968942    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.968991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.968983    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l52fd\" (UniqueName: \"kubernetes.io/projected/7286d5c3-0565-45a2-bc50-20d67023ce69-kube-api-access-l52fd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.968991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.968997    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:07.969250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:07.969009    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7286d5c3-0565-45a2-bc50-20d67023ce69-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:07:08.307808 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:08.307780    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"
Nov 04 00:07:08.308005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:08.307814    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw" event={"ID":"7286d5c3-0565-45a2-bc50-20d67023ce69","Type":"ContainerDied","Data":"b51bae8b161c35610d6efccf5bde7dcc7303d51fd753dd8f7453f125ae9251e1"}
Nov 04 00:07:08.308005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:08.307875    6430 scope.go:122] "RemoveContainer" containerID="23a95723a45269ae4293c85798db374f1383aa98b03a7c5fd37fe9564a64349f"
Nov 04 00:07:08.369024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:08.368985    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 04 00:07:08.371558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:08.371434    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-c6zcw"]
Nov 04 00:07:10.233402 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:10.233352    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" path="/var/lib/kubelet/pods/7286d5c3-0565-45a2-bc50-20d67023ce69/volumes"
Nov 04 00:07:31.293931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:07:31.293886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:07:34.835959 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:07:34.835922    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:08:04.833724 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:08:04.833685    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:08:05.527507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:05.523757    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:08:06.168347 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:06.168306    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 04 00:08:06.169377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:06.169321    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy" containerID="cri-o://2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921" gracePeriod=30
Nov 04 00:08:10.531689 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.531618    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" podUID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" containerName="payload-processing" containerID="cri-o://cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe" gracePeriod=25
Nov 04 00:08:10.820972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.820944    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:08:10.877990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.877939    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\") pod \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") "
Nov 04 00:08:10.878195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.878026    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\") pod \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") "
Nov 04 00:08:10.878195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.878096    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\") pod \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") "
Nov 04 00:08:10.878337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.878300    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\") pod \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\" (UID: \"af94a599-d8ba-42df-a9fc-c0664c5c80ef\") "
Nov 04 00:08:10.878446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.878412    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume" pod "af94a599-d8ba-42df-a9fc-c0664c5c80ef" (UID: "af94a599-d8ba-42df-a9fc-c0664c5c80ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:10.878624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.878590    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume" pod "af94a599-d8ba-42df-a9fc-c0664c5c80ef" (UID: "af94a599-d8ba-42df-a9fc-c0664c5c80ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:10.879125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.879098    6430 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-routing-overlay-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:10.879289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.879128    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af94a599-d8ba-42df-a9fc-c0664c5c80ef-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:10.880811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.880774    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv" pod "af94a599-d8ba-42df-a9fc-c0664c5c80ef" (UID: "af94a599-d8ba-42df-a9fc-c0664c5c80ef"). InnerVolumeSpecName "kube-api-access-kwpzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:10.880928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.880861    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials" pod "af94a599-d8ba-42df-a9fc-c0664c5c80ef" (UID: "af94a599-d8ba-42df-a9fc-c0664c5c80ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:10.921686 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.921653    6430 generic.go:356] "Generic (PLEG): container finished" podID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" containerID="cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe" exitCode=0
Nov 04 00:08:10.921879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.921713    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"
Nov 04 00:08:10.921879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.921727    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" event={"ID":"af94a599-d8ba-42df-a9fc-c0664c5c80ef","Type":"ContainerDied","Data":"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"}
Nov 04 00:08:10.921879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.921793    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2" event={"ID":"af94a599-d8ba-42df-a9fc-c0664c5c80ef","Type":"ContainerDied","Data":"22aaa6787005a632d9adc84d893f742efb1669dd8d0a375f94458f0fae548a5d"}
Nov 04 00:08:10.921879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.921815    6430 scope.go:122] "RemoveContainer" containerID="cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"
Nov 04 00:08:10.936277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.936256    6430 scope.go:122] "RemoveContainer" containerID="cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"
Nov 04 00:08:10.936582 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:08:10.936562    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe\": container with ID starting with cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe not found: ID does not exist" containerID="cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"
Nov 04 00:08:10.936669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.936590    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe"} err="failed to get container status \"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe\": rpc error: code = NotFound desc = could not find container \"cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe\": container with ID starting with cc490925a857a30d2af2275571ee03087545c4cc8fdeaf8aedc7bc4bf47715fe not found: ID does not exist"
Nov 04 00:08:10.988649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.988609    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:08:10.995477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.995366    6430 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-provider-credentials\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:10.995477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:10.995435    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwpzv\" (UniqueName: \"kubernetes.io/projected/af94a599-d8ba-42df-a9fc-c0664c5c80ef-kube-api-access-kwpzv\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.005781 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.001823    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4376d/payload-processing-external-model-e2e-extmod-b-b4376d-858829qf2"]
Nov 04 00:08:11.433068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.433040    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.645966    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646016    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646063    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646165    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646209    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646251    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646306    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646345    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.646522    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\") pod \"b82b45da-12c4-426b-aa07-955f9d903f8d\" (UID: \"b82b45da-12c4-426b-aa07-955f9d903f8d\") "
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.654126    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.654565    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.655163    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.655486    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.660530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.660732    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:11.662625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.660837    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:11.663935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.663171    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "kube-api-access-7hb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:08:11.663935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.663232    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy" pod "b82b45da-12c4-426b-aa07-955f9d903f8d" (UID: "b82b45da-12c4-426b-aa07-955f9d903f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:08:11.748239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748204    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748236    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748247    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748255    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748266    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b82b45da-12c4-426b-aa07-955f9d903f8d-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748277    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748286    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748295    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hb47\" (UniqueName: \"kubernetes.io/projected/b82b45da-12c4-426b-aa07-955f9d903f8d-kube-api-access-7hb47\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.748531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.748303    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b82b45da-12c4-426b-aa07-955f9d903f8d-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:08:11.960467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.959241    6430 generic.go:356] "Generic (PLEG): container finished" podID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerID="2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921" exitCode=0
Nov 04 00:08:11.960467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.959590    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" event={"ID":"b82b45da-12c4-426b-aa07-955f9d903f8d","Type":"ContainerDied","Data":"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"}
Nov 04 00:08:11.960467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.959651    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb" event={"ID":"b82b45da-12c4-426b-aa07-955f9d903f8d","Type":"ContainerDied","Data":"f754cee307489bb15c3ae3d5416dc97fb3ab2b6e6f7d214e3c4e38dc63e28537"}
Nov 04 00:08:11.960467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.959680    6430 scope.go:122] "RemoveContainer" containerID="2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"
Nov 04 00:08:11.960467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:11.959780    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"
Nov 04 00:08:12.046278 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.046248    6430 scope.go:122] "RemoveContainer" containerID="2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"
Nov 04 00:08:12.048797 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:08:12.048760    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921\": container with ID starting with 2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921 not found: ID does not exist" containerID="2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"
Nov 04 00:08:12.048943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.048808    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921"} err="failed to get container status \"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921\": rpc error: code = NotFound desc = could not find container \"2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921\": container with ID starting with 2698c9fe38a98c26ee6bb8fb228cf4e2a31267b22c181f74a75e656d6b56d921 not found: ID does not exist"
Nov 04 00:08:12.050494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.050462    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 04 00:08:12.052912 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.052885    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ppkrb"]
Nov 04 00:08:12.237832 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.237743    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" path="/var/lib/kubelet/pods/af94a599-d8ba-42df-a9fc-c0664c5c80ef/volumes"
Nov 04 00:08:12.238308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:12.238294    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" path="/var/lib/kubelet/pods/b82b45da-12c4-426b-aa07-955f9d903f8d/volumes"
Nov 04 00:08:34.832673 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:08:34.832628    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:08:36.436440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.436383    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:08:36.437372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437348    6430 cpu_manager.go:414] "containerMap: removing container" podUID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" containerName="payload-processing"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437373    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" containerName="payload-processing"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437412    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437420    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437450    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437458    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy"
Nov 04 00:08:36.437521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437515    6430 cpu_manager.go:414] "containerMap: removing container" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437523    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437546    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437554    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437746    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7286d5c3-0565-45a2-bc50-20d67023ce69" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437802    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a1d1ca-ecec-41a0-9642-e41e84f9dd1d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437816    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82b45da-12c4-426b-aa07-955f9d903f8d" containerName="istio-proxy"
Nov 04 00:08:36.437838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437838    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="af94a599-d8ba-42df-a9fc-c0664c5c80ef" containerName="payload-processing"
Nov 04 00:08:36.438190 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.437851    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abf9512-7c01-4031-a73f-f6fae8939179" containerName="istio-proxy"
Nov 04 00:08:36.443509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.443486    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.449779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.449752    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8895cb-openshift-default-dockercfg-lxj7b\""
Nov 04 00:08:36.456283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.456256    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:08:36.478652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.478617    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.478850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.478676    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.478850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.478822    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.478988 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.478925    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.478988 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.478959    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.479112 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.479022    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.479169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.479158    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.479225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.479187    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhmg\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.479299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.479284    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580426    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580479    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580589    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580710    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580740    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhmg\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580905 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580790    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.580948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580921    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.580985    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581070    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581083    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581272 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581250    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581567 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581543    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581643    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.581860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.581838    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.583339 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.583305    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.583523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.583506    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.594053 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.594027    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.594162 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.594146    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhmg\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:36.760546 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:36.760456    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:37.065426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.061540    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:08:37.065426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.061690    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:08:37.065426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.061777    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:08:37.069041 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.069009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:08:37.209889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.209849    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" event={"ID":"472484d2-9979-430b-a985-dfbaf07d3d66","Type":"ContainerStarted","Data":"9f3247f1cb83977f2884ae51b806f9010f1facd722ba6608c22dd0830488fa12"}
Nov 04 00:08:37.210101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.209915    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" event={"ID":"472484d2-9979-430b-a985-dfbaf07d3d66","Type":"ContainerStarted","Data":"c1036251627d62a5a5a9ae3f5d9b072c47992f087db8c9e4d0c98b08431dc063"}
Nov 04 00:08:37.235055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.234993    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podStartSLOduration=1.234979512 podStartE2EDuration="1.234979512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:08:37.233892774 +0000 UTC m=+2333.124954709" watchObservedRunningTime="2026-11-04 00:08:37.234979512 +0000 UTC m=+2333.126041590"
Nov 04 00:08:37.760931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:37.760882    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:38.762105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:38.762071    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 04 00:08:38.762538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:38.762128    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 04 00:08:39.761515 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:39.761479    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 04 00:08:39.761703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:39.761548    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 04 00:08:40.761098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:40.761060    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 04 00:08:40.761646 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:40.761125    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 04 00:08:41.761655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:41.761616    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 04 00:08:41.762063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:41.761673    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 04 00:08:42.761712 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:42.761674    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 04 00:08:42.762203 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:42.761751    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 04 00:08:43.766818 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:43.766783    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:43.767280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:43.767263    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:43.767993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:43.767968    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:08:47.050021 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:08:47.049985    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:09:04.835607 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:09:04.835565    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:09:34.832431 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:09:34.832373    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:09:48.210424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:09:48.210375    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:10:04.833937 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:10:04.833896    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:10:34.837589 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:10:34.837548    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:10:35.624866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:35.624824    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 04 00:10:35.625056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:35.625038    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy" containerID="cri-o://c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea" gracePeriod=30
Nov 04 00:10:40.912737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:40.912705    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 04 00:10:41.028095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028062    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028098    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028279    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028299    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028341    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028426    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:10:41.028665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028449    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:10:41.028665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028528    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\") pod \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\" (UID: \"562e9478-dd97-45bb-8e99-c01fd16b9b5b\") "
Nov 04 00:10:41.028877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028730    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:10:41.028877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028759    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:10:41.028998 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.028661    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:10:41.029306 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.029284    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.029306 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.029305    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.029510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.029315    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.029510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.029327    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.029510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.029340    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.030668 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.030643    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:10:41.030768 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.030646    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:41.031048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.031027    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:10:41.031102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.031048    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277" pod "562e9478-dd97-45bb-8e99-c01fd16b9b5b" (UID: "562e9478-dd97-45bb-8e99-c01fd16b9b5b"). InnerVolumeSpecName "kube-api-access-2h277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:10:41.131241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.131192    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.131241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.131232    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h277\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-kube-api-access-2h277\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.131241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.131250    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.131643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.131261    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562e9478-dd97-45bb-8e99-c01fd16b9b5b-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:10:41.392376 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.392335    6430 generic.go:356] "Generic (PLEG): container finished" podID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerID="c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea" exitCode=0
Nov 04 00:10:41.392581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.392406    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" event={"ID":"562e9478-dd97-45bb-8e99-c01fd16b9b5b","Type":"ContainerDied","Data":"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"}
Nov 04 00:10:41.392581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.392432    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"
Nov 04 00:10:41.392581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.392463    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b" event={"ID":"562e9478-dd97-45bb-8e99-c01fd16b9b5b","Type":"ContainerDied","Data":"c22df7ec35d32448124900a64efc51473057b8b72d2ae55608436159a85c902a"}
Nov 04 00:10:41.392581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.392493    6430 scope.go:122] "RemoveContainer" containerID="c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"
Nov 04 00:10:41.409472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.409435    6430 scope.go:122] "RemoveContainer" containerID="c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"
Nov 04 00:10:41.409886 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:10:41.409862    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea\": container with ID starting with c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea not found: ID does not exist" containerID="c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"
Nov 04 00:10:41.409948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.409897    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea"} err="failed to get container status \"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea\": rpc error: code = NotFound desc = could not find container \"c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea\": container with ID starting with c5fcc588edd895da0962dbcf48765845f942617a0769107331446e775565e5ea not found: ID does not exist"
Nov 04 00:10:41.426210 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.426167    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 04 00:10:41.431077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:41.431041    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-42p4b"]
Nov 04 00:10:42.233340 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:42.233302    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" path="/var/lib/kubelet/pods/562e9478-dd97-45bb-8e99-c01fd16b9b5b/volumes"
Nov 04 00:10:49.709741 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:10:49.709698    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:11:04.831902 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:11:04.831858    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:11:34.833256 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:11:34.833217    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:11:51.559292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.559258    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:11:51.560355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.560331    6430 cpu_manager.go:414] "containerMap: removing container" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy"
Nov 04 00:11:51.560355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.560354    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy"
Nov 04 00:11:51.560621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.560602    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="562e9478-dd97-45bb-8e99-c01fd16b9b5b" containerName="istio-proxy"
Nov 04 00:11:51.565044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.565020    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.584600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.584569    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:11:51.597617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.597588    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:11:51.597816 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.597792    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.609527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.609485    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:11:51.683481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683443    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683569    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683630    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.683661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683650    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683760    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683811    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.683877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683845    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683908    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xml\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.683967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683951    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.683975    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.684100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684078    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.684141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684127    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684165    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684211    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684294 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684269    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvcr\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684333    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.684498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684354    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.684498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.684372    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.785684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785649    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.785684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785689    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.785915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785759    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.785915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785811    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.785915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785858    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.785915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785888    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xml\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785927    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.785957    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786036    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786087    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786110    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786155    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786190    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvcr\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786207    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786226    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786259    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786290    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786329    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786328    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786515    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786580    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786633    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.786677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.787008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786723    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.787008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.786909    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.787161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.787136    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.787443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.787381    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.787582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.787462    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.788886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.788863    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.789011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.788862    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.789241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.789220    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.789330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.789314    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.798585 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.798564    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.799002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.798982    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xml\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q9pm8\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:51.800776 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.800758    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvcr\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.800958 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.800942    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-xtggb\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.887941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.887850    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:51.919287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:51.919259    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:52.510496 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.510467    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:11:52.511178 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:11:52.511153    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43df75bf_6403_45f4_87e6_844f1b009a06.slice/crio-81d2c506386fa9516a25f66a15b118497a91ab6a5c0aec1945d77e2004e14d9b WatchSource:0}: Error finding container 81d2c506386fa9516a25f66a15b118497a91ab6a5c0aec1945d77e2004e14d9b: Status 404 returned error can't find the container with id 81d2c506386fa9516a25f66a15b118497a91ab6a5c0aec1945d77e2004e14d9b
Nov 04 00:11:52.514398 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.514325    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:52.514476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.514430    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:52.514476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.514466    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:52.577331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.574692    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:11:52.586016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.585455    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:52.586016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.585586    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:52.586016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:52.585663    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:11:53.370541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.365824    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" event={"ID":"1226101c-28de-4eed-a854-c6db2a7b8540","Type":"ContainerStarted","Data":"b49c76bf77a085fd930b5ec31ac6c10d6d177ab37deb1d881a70d5886058c658"}
Nov 04 00:11:53.370541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.365908    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" event={"ID":"1226101c-28de-4eed-a854-c6db2a7b8540","Type":"ContainerStarted","Data":"5d841386d98004c5e401df42dea35ec7fed1790541626c721ccd7f8f449777a5"}
Nov 04 00:11:53.396449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.394327    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" event={"ID":"43df75bf-6403-45f4-87e6-844f1b009a06","Type":"ContainerStarted","Data":"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"}
Nov 04 00:11:53.396449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.394417    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" event={"ID":"43df75bf-6403-45f4-87e6-844f1b009a06","Type":"ContainerStarted","Data":"81d2c506386fa9516a25f66a15b118497a91ab6a5c0aec1945d77e2004e14d9b"}
Nov 04 00:11:53.416420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.415464    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podStartSLOduration=2.415444725 podStartE2EDuration="2.415444725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:53.411125814 +0000 UTC m=+2529.302187752" watchObservedRunningTime="2026-11-04 00:11:53.415444725 +0000 UTC m=+2529.306506664"
Nov 04 00:11:53.445551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.444250    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podStartSLOduration=2.444229867 podStartE2EDuration="2.444229867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:53.441910487 +0000 UTC m=+2529.332972426" watchObservedRunningTime="2026-11-04 00:11:53.444229867 +0000 UTC m=+2529.335291803"
Nov 04 00:11:53.888691 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.888652    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:11:53.890066 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.890038    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:53.890209 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.890090    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:53.923270 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.923168    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:11:53.924478 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.924445    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:53.924699 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:53.924675    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:54.888936 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:54.888901    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:54.889453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:54.889380    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:54.931459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:54.926786    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:54.931459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:54.926849    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:55.888376 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:55.888336    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:55.888611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:55.888421    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:55.957132 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:55.957092    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:55.957820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:55.957786    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:56.888695 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:56.888652    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:56.888891 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:56.888711    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:56.922882 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:56.922838    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:56.923205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:56.923164    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:57.888934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:57.888895    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:57.889613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:57.888959    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:57.927421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:57.926927    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:57.927421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:57.926993    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:58.888289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:58.888251    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:58.888518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:58.888309    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:58.931415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:58.928995    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:58.931415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:58.929080    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:11:59.889151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:59.889103    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:11:59.889381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:59.889159    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:11:59.920728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:59.920691    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 04 00:11:59.920906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:11:59.920749    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 04 00:12:00.145521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:00.145437    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:12:00.888168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:00.888129    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 04 00:12:00.888424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:00.888186    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 04 00:12:01.962556 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:01.962497    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:01.963081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:01.963046    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:02.893346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:02.893301    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:02.893708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:02.893671    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:02.963683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:02.963639    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:02.964195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:02.963774    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:03.947256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:03.946493    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:03.947256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:03.946566    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:04.009232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:04.008532    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:04.009232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:04.008613    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:05.010414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:05.007354    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:05.010414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:05.007466    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:05.079463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:05.072117    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:05.079463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:05.072247    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:05.282468 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:12:05.281849    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:12:06.011420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.007953    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:06.011420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.008028    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:06.113605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.113562    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:06.113791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.113642    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:06.201967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.201908    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:12:06.202847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.202813    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:12:06.203800 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.203770    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:12:06.417469 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.416596    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:12:06.418063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.418037    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:12:06.419269 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:06.419239    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:12:27.309054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:27.309003    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-km48c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:12:27.309688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:27.309088    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:12:34.831527 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:12:34.831491    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:12:49.488755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:12:49.488715    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:13:04.832877 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:13:04.832839    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:13:34.832737 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:13:34.832698    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:13:48.060673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:13:48.060632    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:13:52.257509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:13:52.257473    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-xtggb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:13:52.258119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:13:52.258087    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:13:52.504509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:13:52.504467    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q9pm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:13:52.504870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:13:52.504839    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:14:04.833710 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:14:04.833666    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:14:34.832777 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:14:34.832738    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:14:43.395810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:43.395774    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:14:52.915786 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:52.915746    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 04 00:14:52.916382 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:52.916011    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy" containerID="cri-o://98aa226e2bf2ce341174c79676badd9dd1b6efcfb6c8f3f5fb14a70afddf645a" gracePeriod=30
Nov 04 00:14:58.042846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.042817    6430 generic.go:356] "Generic (PLEG): container finished" podID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerID="98aa226e2bf2ce341174c79676badd9dd1b6efcfb6c8f3f5fb14a70afddf645a" exitCode=0
Nov 04 00:14:58.043260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.042890    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerDied","Data":"98aa226e2bf2ce341174c79676badd9dd1b6efcfb6c8f3f5fb14a70afddf645a"}
Nov 04 00:14:58.043260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.042949    6430 scope.go:122] "RemoveContainer" containerID="b828cb530ef173a45db5398a5088d3dc061579fe2084445e6b7a650e46d885fc"
Nov 04 00:14:58.215834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.215796    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 04 00:14:58.312775 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.312741    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.312968 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.312805    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.312968 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.312835    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.312968 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.312937    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313031    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313071    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313134    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:14:58.313218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313172    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313189    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:14:58.313218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313215    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313495 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313314    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\") pod \"69e74a34-036b-4930-b8df-4219c4cfb09c\" (UID: \"69e74a34-036b-4930-b8df-4219c4cfb09c\") "
Nov 04 00:14:58.313495 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313482    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:14:58.313619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313513    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:14:58.313788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.313764    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:14:58.314220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.314193    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.314220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.314223    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.314455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.314238    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e74a34-036b-4930-b8df-4219c4cfb09c-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.314455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.314252    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.314455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.314262    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.315620 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.315597    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:14:58.315620 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.315600    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:58.315758 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.315730    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:14:58.315814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.315796    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4" pod "69e74a34-036b-4930-b8df-4219c4cfb09c" (UID: "69e74a34-036b-4930-b8df-4219c4cfb09c"). InnerVolumeSpecName "kube-api-access-kccc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:14:58.415862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.415821    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.415862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.415852    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.415862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.415864    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kccc4\" (UniqueName: \"kubernetes.io/projected/69e74a34-036b-4930-b8df-4219c4cfb09c-kube-api-access-kccc4\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:58.415862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:58.415873    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e74a34-036b-4930-b8df-4219c4cfb09c-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:14:59.050545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:59.050511    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"
Nov 04 00:14:59.051014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:59.050547    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb" event={"ID":"69e74a34-036b-4930-b8df-4219c4cfb09c","Type":"ContainerDied","Data":"799eea3db3b8c735c27a4311eafa43f5254d31b48a3bb4fb26fb37dba7cd8b3e"}
Nov 04 00:14:59.051014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:59.050608    6430 scope.go:122] "RemoveContainer" containerID="98aa226e2bf2ce341174c79676badd9dd1b6efcfb6c8f3f5fb14a70afddf645a"
Nov 04 00:14:59.077059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:59.077021    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 04 00:14:59.080228 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:14:59.080201    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-5twdb"]
Nov 04 00:15:00.243402 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.243349    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" path="/var/lib/kubelet/pods/69e74a34-036b-4930-b8df-4219c4cfb09c/volumes"
Nov 04 00:15:00.244830 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.244760    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:00.248892 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.248860    6430 cpu_manager.go:414] "containerMap: removing container" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.249048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.249029    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.249152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.249140    6430 cpu_manager.go:414] "containerMap: removing container" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.249240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.249229    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.250015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.249996    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.250147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.250134    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e74a34-036b-4930-b8df-4219c4cfb09c" containerName="istio-proxy"
Nov 04 00:15:00.260234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.260199    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:00.260460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.260444    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:00.260607 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.260328    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.269701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.269665    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.277178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.277149    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:00.350043 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.350001    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.350261 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.350058    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtfb\" (UniqueName: \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.350446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.350380    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.350552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.350485    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8km\" (UniqueName: \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.354519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.354494    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:15:00.452381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.452339    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.452592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.452441    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8km\" (UniqueName: \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.452673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.452612    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.452673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.452661    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtfb\" (UniqueName: \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.453194 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.453146    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.453450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.453423    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.465793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.465769    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtfb\" (UniqueName: \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.465793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.465781    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8km\" (UniqueName: \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\") pod \"maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.589400 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.589363    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:00.596015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.595993    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:00.878451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.878421    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:00.878791 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:15:00.878755    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b3ce0c_df43_48fb_8ec2_5ccfe5fee934.slice/crio-d1c2644acb27179d059b9f31fe5f65641598791372a84514ffedef140d61b174 WatchSource:0}: Error finding container d1c2644acb27179d059b9f31fe5f65641598791372a84514ffedef140d61b174: Status 404 returned error can't find the container with id d1c2644acb27179d059b9f31fe5f65641598791372a84514ffedef140d61b174
Nov 04 00:15:00.885436 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:15:00.885365    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c01aad_4091_4c82_8cbf_d780b2f4b49e.slice/crio-38f46cf285e705787619065fbadfbb0d95c0d000bf6b387b63f58552086fe263 WatchSource:0}: Error finding container 38f46cf285e705787619065fbadfbb0d95c0d000bf6b387b63f58552086fe263: Status 404 returned error can't find the container with id 38f46cf285e705787619065fbadfbb0d95c0d000bf6b387b63f58552086fe263
Nov 04 00:15:00.885747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:00.885722    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:01.070498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:01.070456    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerStarted","Data":"38f46cf285e705787619065fbadfbb0d95c0d000bf6b387b63f58552086fe263"}
Nov 04 00:15:01.071843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:01.071817    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerStarted","Data":"d1c2644acb27179d059b9f31fe5f65641598791372a84514ffedef140d61b174"}
Nov 04 00:15:02.081898 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.081824    6430 generic.go:356] "Generic (PLEG): container finished" podID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerID="472efdb39ed9f54dc2d8ce8af43676b0d768e342c9964b0311922135d1e27eeb" exitCode=60
Nov 04 00:15:02.082324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.081919    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerDied","Data":"472efdb39ed9f54dc2d8ce8af43676b0d768e342c9964b0311922135d1e27eeb"}
Nov 04 00:15:02.082324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.082156    6430 scope.go:122] "RemoveContainer" containerID="472efdb39ed9f54dc2d8ce8af43676b0d768e342c9964b0311922135d1e27eeb"
Nov 04 00:15:02.084035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.083911    6430 generic.go:356] "Generic (PLEG): container finished" podID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerID="aaba9995b88545b39b69499413b05b57f9b080f98cfa07b0920761b8181bee23" exitCode=60
Nov 04 00:15:02.084035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.083983    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerDied","Data":"aaba9995b88545b39b69499413b05b57f9b080f98cfa07b0920761b8181bee23"}
Nov 04 00:15:02.084510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:02.084316    6430 scope.go:122] "RemoveContainer" containerID="aaba9995b88545b39b69499413b05b57f9b080f98cfa07b0920761b8181bee23"
Nov 04 00:15:03.096241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:03.096203    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerStarted","Data":"e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"}
Nov 04 00:15:03.098297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:03.098270    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerStarted","Data":"a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"}
Nov 04 00:15:03.120869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:03.120801    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" podStartSLOduration=3.120781007 podStartE2EDuration="3.120781007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.120488322 +0000 UTC m=+2719.011550258" watchObservedRunningTime="2026-11-04 00:15:03.120781007 +0000 UTC m=+2719.011842944"
Nov 04 00:15:03.139486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:03.139369    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" podStartSLOduration=3.139351464 podStartE2EDuration="3.139351464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.139276781 +0000 UTC m=+2719.030338740" watchObservedRunningTime="2026-11-04 00:15:03.139351464 +0000 UTC m=+2719.030413394"
Nov 04 00:15:04.109295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.109262    6430 generic.go:356] "Generic (PLEG): container finished" podID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerID="e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08" exitCode=60
Nov 04 00:15:04.109757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.109341    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerDied","Data":"e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"}
Nov 04 00:15:04.109757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.109427    6430 scope.go:122] "RemoveContainer" containerID="aaba9995b88545b39b69499413b05b57f9b080f98cfa07b0920761b8181bee23"
Nov 04 00:15:04.109757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.109725    6430 scope.go:122] "RemoveContainer" containerID="e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"
Nov 04 00:15:04.110098 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:04.110080    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc_odh-ai-gateway-infra(47c01aad-4091-4c82-8cbf-d780b2f4b49e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e"
Nov 04 00:15:04.119406 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.112449    6430 generic.go:356] "Generic (PLEG): container finished" podID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerID="a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8" exitCode=60
Nov 04 00:15:04.119406 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.112580    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerDied","Data":"a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"}
Nov 04 00:15:04.119406 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.113123    6430 scope.go:122] "RemoveContainer" containerID="a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"
Nov 04 00:15:04.119406 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:04.113704    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr_odh-ai-gateway-infra(66b3ce0c-df43-48fb-8ec2-5ccfe5fee934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934"
Nov 04 00:15:04.136310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:04.136285    6430 scope.go:122] "RemoveContainer" containerID="472efdb39ed9f54dc2d8ce8af43676b0d768e342c9964b0311922135d1e27eeb"
Nov 04 00:15:04.833354 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:04.833318    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:15:05.124181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:05.124083    6430 scope.go:122] "RemoveContainer" containerID="a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"
Nov 04 00:15:05.124672 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:05.124568    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr_odh-ai-gateway-infra(66b3ce0c-df43-48fb-8ec2-5ccfe5fee934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934"
Nov 04 00:15:05.125977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:05.125946    6430 scope.go:122] "RemoveContainer" containerID="e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"
Nov 04 00:15:05.126295 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:05.126270    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc_odh-ai-gateway-infra(47c01aad-4091-4c82-8cbf-d780b2f4b49e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e"
Nov 04 00:15:07.776568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.776531    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:15:07.791644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.791609    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:15:07.791899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.791729    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.797132 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.797108    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-11301e-openshift-default-dockercfg-gv2bz\""
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.969502    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.969571    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.969659    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.969840    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.969916    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.970057    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.970147    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxw6x\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.970219    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:07.983484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:07.970248    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072543    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072729    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072765    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072800    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.072941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.073004    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.073122    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.073503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.073203    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxw6x\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.075992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.074893    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.075992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.075152    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.081504 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.077588    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.082551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.082519    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.082701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.075377    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.106032 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.105718    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.114094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.110717    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.117472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.114573    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.117472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.116065    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxw6x\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\") pod \"e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") " pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.136875 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.136774    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:08.488863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.488830    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:15:08.489693 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:15:08.489668    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32980f4c_0f70_4230_b077_2d2746a25cda.slice/crio-9416c475365ff878d047682ae5131fd21a66156a3690bc1a5f3c16733c77368e WatchSource:0}: Error finding container 9416c475365ff878d047682ae5131fd21a66156a3690bc1a5f3c16733c77368e: Status 404 returned error can't find the container with id 9416c475365ff878d047682ae5131fd21a66156a3690bc1a5f3c16733c77368e
Nov 04 00:15:08.492673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.492642    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:15:08.492761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.492711    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:15:08.492761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:08.492742    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:15:09.167902 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:09.167861    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerStarted","Data":"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"}
Nov 04 00:15:09.168473 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:09.167926    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerStarted","Data":"9416c475365ff878d047682ae5131fd21a66156a3690bc1a5f3c16733c77368e"}
Nov 04 00:15:09.192357 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:09.192299    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podStartSLOduration=2.19228695 podStartE2EDuration="2.19228695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:09.191743548 +0000 UTC m=+2725.082805500" watchObservedRunningTime="2026-11-04 00:15:09.19228695 +0000 UTC m=+2725.083348885"
Nov 04 00:15:10.137683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:10.137649    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:10.144219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:10.144191    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:10.175182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:10.175152    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:10.176088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:10.176070    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:15:16.224772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:16.224741    6430 scope.go:122] "RemoveContainer" containerID="a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"
Nov 04 00:15:17.225009 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:17.224979    6430 scope.go:122] "RemoveContainer" containerID="e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"
Nov 04 00:15:17.302031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:17.301994    6430 generic.go:356] "Generic (PLEG): container finished" podID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerID="1029b82bd94de0d1facb10c528f0ffca776678c0d1a29cf4ee8f915ad74c3c5d" exitCode=60
Nov 04 00:15:17.302233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:17.302058    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerDied","Data":"1029b82bd94de0d1facb10c528f0ffca776678c0d1a29cf4ee8f915ad74c3c5d"}
Nov 04 00:15:17.302233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:17.302116    6430 scope.go:122] "RemoveContainer" containerID="a09a0d527698dff9e8a12b1443f9030c52f7c4661b504fc5eaf248e1e985c8e8"
Nov 04 00:15:17.302494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:17.302475    6430 scope.go:122] "RemoveContainer" containerID="1029b82bd94de0d1facb10c528f0ffca776678c0d1a29cf4ee8f915ad74c3c5d"
Nov 04 00:15:17.302777 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:17.302753    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr_odh-ai-gateway-infra(66b3ce0c-df43-48fb-8ec2-5ccfe5fee934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934"
Nov 04 00:15:18.312348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.312211    6430 generic.go:356] "Generic (PLEG): container finished" podID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerID="7e15ea3e26bb5af262fd25cba15e615338d93ccc18cd97b5a95dfd67a728f3a2" exitCode=60
Nov 04 00:15:18.312836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.312278    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerDied","Data":"7e15ea3e26bb5af262fd25cba15e615338d93ccc18cd97b5a95dfd67a728f3a2"}
Nov 04 00:15:18.312836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.312464    6430 scope.go:122] "RemoveContainer" containerID="e2e0aca26e9f5446d93498c9a63577b03738cb08d0a72d04ba77edcac6640d08"
Nov 04 00:15:18.312973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.312879    6430 scope.go:122] "RemoveContainer" containerID="7e15ea3e26bb5af262fd25cba15e615338d93ccc18cd97b5a95dfd67a728f3a2"
Nov 04 00:15:18.313348 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:18.313317    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc_odh-ai-gateway-infra(47c01aad-4091-4c82-8cbf-d780b2f4b49e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e"
Nov 04 00:15:18.351527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.339438    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:18.545896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.545870    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:18.618829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.618787    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\" (UniqueName: \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\") pod \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") "
Nov 04 00:15:18.619020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.618888    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\") pod \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\" (UID: \"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934\") "
Nov 04 00:15:18.620081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.620036    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca" pod "66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" (UID: "66b3ce0c-df43-48fb-8ec2-5ccfe5fee934"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:18.623545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.623507    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb" pod "66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" (UID: "66b3ce0c-df43-48fb-8ec2-5ccfe5fee934"). InnerVolumeSpecName "kube-api-access-kwtfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:18.720802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.720765    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:18.720802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:18.720795    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtfb\" (UniqueName: \"kubernetes.io/projected/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934-kube-api-access-kwtfb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:19.325599 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.325567    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"
Nov 04 00:15:19.326025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.325569    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr" event={"ID":"66b3ce0c-df43-48fb-8ec2-5ccfe5fee934","Type":"ContainerDied","Data":"d1c2644acb27179d059b9f31fe5f65641598791372a84514ffedef140d61b174"}
Nov 04 00:15:19.326025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.325705    6430 scope.go:122] "RemoveContainer" containerID="1029b82bd94de0d1facb10c528f0ffca776678c0d1a29cf4ee8f915ad74c3c5d"
Nov 04 00:15:19.347247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.347219    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:19.357427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.357381    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:19.363588 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.363561    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7d218-29895855-729kr"]
Nov 04 00:15:19.595243 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.595180    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:19.646553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.646521    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\" (UniqueName: \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\") pod \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") "
Nov 04 00:15:19.646762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.646703    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\") pod \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\" (UID: \"47c01aad-4091-4c82-8cbf-d780b2f4b49e\") "
Nov 04 00:15:19.647074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.647051    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca" pod "47c01aad-4091-4c82-8cbf-d780b2f4b49e" (UID: "47c01aad-4091-4c82-8cbf-d780b2f4b49e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:15:19.647352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.647335    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c01aad-4091-4c82-8cbf-d780b2f4b49e-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:19.648932 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.648910    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km" pod "47c01aad-4091-4c82-8cbf-d780b2f4b49e" (UID: "47c01aad-4091-4c82-8cbf-d780b2f4b49e"). InnerVolumeSpecName "kube-api-access-bm8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:15:19.748440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:19.748368    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm8km\" (UniqueName: \"kubernetes.io/projected/47c01aad-4091-4c82-8cbf-d780b2f4b49e-kube-api-access-bm8km\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:15:20.234287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.234252    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" path="/var/lib/kubelet/pods/66b3ce0c-df43-48fb-8ec2-5ccfe5fee934/volumes"
Nov 04 00:15:20.334200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.334163    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"
Nov 04 00:15:20.334200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.334193    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc" event={"ID":"47c01aad-4091-4c82-8cbf-d780b2f4b49e","Type":"ContainerDied","Data":"38f46cf285e705787619065fbadfbb0d95c0d000bf6b387b63f58552086fe263"}
Nov 04 00:15:20.334763 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.334269    6430 scope.go:122] "RemoveContainer" containerID="7e15ea3e26bb5af262fd25cba15e615338d93ccc18cd97b5a95dfd67a728f3a2"
Nov 04 00:15:20.357384 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.357354    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:20.362942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:20.362916    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fa0d0a-29895855-gjrqc"]
Nov 04 00:15:22.239006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:22.238973    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" path="/var/lib/kubelet/pods/47c01aad-4091-4c82-8cbf-d780b2f4b49e/volumes"
Nov 04 00:15:34.859607 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:15:34.859566    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:15:35.471140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471101    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:15:35.471858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471842    6430 cpu_manager.go:414] "containerMap: removing container" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471859    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471870    6430 cpu_manager.go:414] "containerMap: removing container" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471877    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471893    6430 cpu_manager.go:414] "containerMap: removing container" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471898    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471912    6430 cpu_manager.go:414] "containerMap: removing container" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.471920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471916    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471934    6430 cpu_manager.go:414] "containerMap: removing container" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471939    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471953    6430 cpu_manager.go:414] "containerMap: removing container" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.471957    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.472062    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.472072    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.472081    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c01aad-4091-4c82-8cbf-d780b2f4b49e" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.472093    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.472152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.472098    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:15:35.476114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.476082    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.485063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.485036    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-11301e-serving-cert\""
Nov 04 00:15:35.489309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.489273    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:15:35.637875 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.637824    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.638165 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.638146    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.638527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.638510    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.638715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.638689    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77t7b\" (UniqueName: \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.740457 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.740338    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.740457 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.740373    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.740717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.740502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.740717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.740538    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77t7b\" (UniqueName: \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.741086 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.741059    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.743005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.742982    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.743174 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.743151    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.753536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.753513    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77t7b\" (UniqueName: \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\") pod \"maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:35.812760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:35.812718    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:36.100888 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:15:36.100854    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c09bed_5c66_460e_8deb_fc154987fa9c.slice/crio-50bfcbb4bca4d4226a12c2924b8b04a0d51c091345f29ba98e2206e39a937da4 WatchSource:0}: Error finding container 50bfcbb4bca4d4226a12c2924b8b04a0d51c091345f29ba98e2206e39a937da4: Status 404 returned error can't find the container with id 50bfcbb4bca4d4226a12c2924b8b04a0d51c091345f29ba98e2206e39a937da4
Nov 04 00:15:36.102206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:36.102187    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:15:36.105127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:36.105098    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:15:36.651416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:36.650570    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" event={"ID":"45c09bed-5c66-460e-8deb-fc154987fa9c","Type":"ContainerStarted","Data":"50bfcbb4bca4d4226a12c2924b8b04a0d51c091345f29ba98e2206e39a937da4"}
Nov 04 00:15:37.659442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:37.659382    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" event={"ID":"45c09bed-5c66-460e-8deb-fc154987fa9c","Type":"ContainerStarted","Data":"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"}
Nov 04 00:15:37.659941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:37.659472    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:15:37.683764 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:37.683713    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" podStartSLOduration=2.015858977 podStartE2EDuration="2.683698652s" totalImagesPullingTime="667.839675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:37.680736073 +0000 UTC m=+2753.571798009" watchObservedRunningTime="2026-11-04 00:15:37.683698652 +0000 UTC m=+2753.574760586"
Nov 04 00:15:43.672566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:15:43.672531    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:16:02.137260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:02.137224    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:16:04.860305 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:16:04.860259    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:16:11.216990 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:11.216946    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:16:11.217652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:11.217619    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:16:16.307076 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.307040    6430 generic.go:356] "Generic (PLEG): container finished" podID="32980f4c-0f70-4230-b077-2d2746a25cda" containerID="c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375" exitCode=0
Nov 04 00:16:16.307560 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.307109    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerDied","Data":"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"}
Nov 04 00:16:16.307623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.307608    6430 scope.go:122] "RemoveContainer" containerID="c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"
Nov 04 00:16:16.309826 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.309799    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:16.309934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.309869    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:16.309934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:16.309901    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:17.353718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:17.353677    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerStarted","Data":"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"}
Nov 04 00:16:18.138620 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:18.138440    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:16:18.138825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:18.138658    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:18.138825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:18.138722    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:19.137989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:19.137956    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:19.138443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:19.138008    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:20.138079 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:20.138043    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:20.138526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:20.138111    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:21.138592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.138553    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:21.139230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.139181    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:21.605411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.600326    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:16:21.612103 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.612069    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b3ce0c-df43-48fb-8ec2-5ccfe5fee934" containerName="cleanup"
Nov 04 00:16:21.660759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.650146    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.719416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.663477    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:16:21.845590 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.750129    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.845899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.845876    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.846082 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.846067    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.846209 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.846196    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.846316 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.846304    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.855561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.800500    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9a6f7a-openshift-default-dockercfg-m7fs7\""
Nov 04 00:16:21.873550 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.873512    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.873802 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.873784    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.874048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.874026    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54fp\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.874314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.874289    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.978763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.978834    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.978910    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.978956    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.978992    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.979051    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.979081    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.979175    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54fp\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:21.989600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:21.979300    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.011636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.004126    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.011636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.004228    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.011636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.004522    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.011636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.006181    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.040719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.039759    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.040719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.039983    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.044022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.043964    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.053022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.052814    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.058052 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.055804    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54fp\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.251181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.251085    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:22.254874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.254840    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:22.330410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.313832    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:22.635337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.635294    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:16:22.637599 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:16:22.637559    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe02de0_d5c4_4a25_8a66_73ea81469082.slice/crio-149d546efea9d453a2a8c82fdb45ec36fd65e677e5a466515aa16b31038cd6ad WatchSource:0}: Error finding container 149d546efea9d453a2a8c82fdb45ec36fd65e677e5a466515aa16b31038cd6ad: Status 404 returned error can't find the container with id 149d546efea9d453a2a8c82fdb45ec36fd65e677e5a466515aa16b31038cd6ad
Nov 04 00:16:22.641178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.641138    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:22.641297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.641231    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:22.641297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:22.641268    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:16:23.139246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:23.139214    6430 patch_prober.go:29] interesting pod/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 04 00:16:23.139479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:23.139268    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 04 00:16:23.464500 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:23.464370    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerStarted","Data":"2c451b60d128f807b5d52130b903d47cf90d081b8013318ad258af2204ae99e3"}
Nov 04 00:16:23.464500 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:23.464487    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerStarted","Data":"149d546efea9d453a2a8c82fdb45ec36fd65e677e5a466515aa16b31038cd6ad"}
Nov 04 00:16:23.494482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:23.494425    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podStartSLOduration=2.494410582 podStartE2EDuration="2.494410582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:23.490374665 +0000 UTC m=+2799.381436608" watchObservedRunningTime="2026-11-04 00:16:23.494410582 +0000 UTC m=+2799.385472514"
Nov 04 00:16:24.142893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.142861    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:16:24.143226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.143204    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:16:24.143983 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.143962    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:16:24.316111 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.316039    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:24.323094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.323066    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:24.472219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.472129    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:24.472959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:24.472942    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:16:34.832725 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:16:34.832682    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:16:43.701422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:43.701369    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:16:48.954666 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:48.954630    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:16:48.959779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:48.959753    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:48.968823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:48.968796    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9a6f7a-serving-cert\""
Nov 04 00:16:48.977806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:48.977729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:16:49.133413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.131014    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zs2h\" (UniqueName: \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.133413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.131195    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.133413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.131339    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.133413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.131525    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.233216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.233135    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.233370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.233231    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.233370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.233295    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zs2h\" (UniqueName: \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.233516 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.233379    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.233966 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.233941    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.235835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.235813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.235911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.235813    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.246380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.246352    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zs2h\" (UniqueName: \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\") pod \"maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.283605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.283572    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:49.612600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.602578    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:16:49.832580 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:49.832541    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" event={"ID":"4d04e01a-1e9e-41d1-9ae1-48be623223e0","Type":"ContainerStarted","Data":"30ff0c30bbb5a9bd47824943dac208634554ab3d876f0e5866ce2d21b2497057"}
Nov 04 00:16:50.842146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:50.841939    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" event={"ID":"4d04e01a-1e9e-41d1-9ae1-48be623223e0","Type":"ContainerStarted","Data":"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"}
Nov 04 00:16:50.842146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:50.842091    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:16:50.865249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:50.865197    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" podStartSLOduration=2.340250915 podStartE2EDuration="2.865182249s" totalImagesPullingTime="524.931334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:50.863780782 +0000 UTC m=+2826.754842738" watchObservedRunningTime="2026-11-04 00:16:50.865182249 +0000 UTC m=+2826.756244216"
Nov 04 00:16:56.892412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:16:56.886220    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:17:04.833033 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:17:04.832996    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:17:20.242144 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.242107    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:17:20.251342 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.249412    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.252573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.252543    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:17:20.258551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.258525    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8cc1c0\"/\"kube-root-ca.crt\""
Nov 04 00:17:20.258759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.258742    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8cc1c0\"/\"openshift-service-ca.crt\""
Nov 04 00:17:20.258833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.258753    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8cc1c0\"/\"default-dockercfg-lxwmv\""
Nov 04 00:17:20.258833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.258771    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8cc1c0\"/\"test-model-8cc1c0-kserve-self-signed-certs\""
Nov 04 00:17:20.366720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.366688    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.366915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.366726    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99lq\" (UniqueName: \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.366915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.366756    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.367019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.366950    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.367019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.366996    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.367131 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.367118    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.472862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.472822    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.473216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.473188    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.473369 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.473349    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99lq\" (UniqueName: \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.473536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.473519    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.475454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.473807    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.475693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.475669    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.476931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.476909    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.477848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.477228    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.477994 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.477558    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.481815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.481793    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.493982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.493920    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.517456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.504931    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99lq\" (UniqueName: \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\") pod \"test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") " pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.572736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.572703    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:20.882781 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:17:20.882749    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafb79de9_ab1f_44d9_a222_2642eca7bea9.slice/crio-b6e520ff592c0c63a15ff31ff4db5a927fee46fe6f7761a56ae465236d16052c WatchSource:0}: Error finding container b6e520ff592c0c63a15ff31ff4db5a927fee46fe6f7761a56ae465236d16052c: Status 404 returned error can't find the container with id b6e520ff592c0c63a15ff31ff4db5a927fee46fe6f7761a56ae465236d16052c
Nov 04 00:17:20.882929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:20.882884    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:17:21.203629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:21.203535    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" event={"ID":"afb79de9-ab1f-44d9-a222-2642eca7bea9","Type":"ContainerStarted","Data":"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"}
Nov 04 00:17:21.203629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:21.203584    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" event={"ID":"afb79de9-ab1f-44d9-a222-2642eca7bea9","Type":"ContainerStarted","Data":"b6e520ff592c0c63a15ff31ff4db5a927fee46fe6f7761a56ae465236d16052c"}
Nov 04 00:17:21.227663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:21.227601    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" podStartSLOduration=1.227582396 podStartE2EDuration="1.227582396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:21.225318947 +0000 UTC m=+2857.116380884" watchObservedRunningTime="2026-11-04 00:17:21.227582396 +0000 UTC m=+2857.118644333"
Nov 04 00:17:21.581453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:21.580151    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:30.574366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.574328    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:30.588353 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.588321    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:30.601596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.601566    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:17:30.968783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.960120    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:17:30.968783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.966599    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:30.975184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.974678    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a7cbc3-openshift-default-dockercfg-tbcj5\""
Nov 04 00:17:30.975638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:30.975613    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:17:31.126420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126099    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d974s\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126209    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126240    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126356    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126461    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126501    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126622    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126673    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.126803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.126740    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.229851    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d974s\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.229939    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.229967    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230050    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230111    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230144    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230245    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230287    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230342    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.230996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.230836    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.236416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.231421    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.236416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.232065    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.236416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.233994    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.236416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.234492    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.296153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.272994    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.296153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.275780    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d974s\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.296153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.285719    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.312281 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.308952    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\") pod \"e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") " pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.598147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.598107    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:31.973989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.973928    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:17:31.979309 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:17:31.979252    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a2ad27_1b90_42ff_8977_abb226bc11c3.slice/crio-a758fb632595688e5e265e40123195880030fc0efe7a2205b4a1ecd8ae19b6d6 WatchSource:0}: Error finding container a758fb632595688e5e265e40123195880030fc0efe7a2205b4a1ecd8ae19b6d6: Status 404 returned error can't find the container with id a758fb632595688e5e265e40123195880030fc0efe7a2205b4a1ecd8ae19b6d6
Nov 04 00:17:31.985067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.985009    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:31.985244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.985109    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:31.985244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:31.985150    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:32.654423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:32.654377    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerStarted","Data":"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"}
Nov 04 00:17:32.654850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:32.654447    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerStarted","Data":"a758fb632595688e5e265e40123195880030fc0efe7a2205b4a1ecd8ae19b6d6"}
Nov 04 00:17:32.683755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:32.683705    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podStartSLOduration=2.6836894 podStartE2EDuration="2.6836894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:32.680582441 +0000 UTC m=+2868.571644380" watchObservedRunningTime="2026-11-04 00:17:32.6836894 +0000 UTC m=+2868.574751329"
Nov 04 00:17:33.609655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:33.609611    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:33.616556 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:33.616525    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:33.663116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:33.663075    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:33.664186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:33.664163    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:17:34.673200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.673162    6430 generic.go:356] "Generic (PLEG): container finished" podID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerID="2c451b60d128f807b5d52130b903d47cf90d081b8013318ad258af2204ae99e3" exitCode=0
Nov 04 00:17:34.673794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.673236    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerDied","Data":"2c451b60d128f807b5d52130b903d47cf90d081b8013318ad258af2204ae99e3"}
Nov 04 00:17:34.673794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.673778    6430 scope.go:122] "RemoveContainer" containerID="2c451b60d128f807b5d52130b903d47cf90d081b8013318ad258af2204ae99e3"
Nov 04 00:17:34.675897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.675870    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:34.675969 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.675952    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:34.676005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:34.675985    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:17:34.845039 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:17:34.845006    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:17:35.682083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:35.682044    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerStarted","Data":"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"}
Nov 04 00:17:36.342611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:36.340266    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:17:36.342977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:36.342948    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 04 00:17:36.343187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:36.343160    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 04 00:17:37.331456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:37.328485    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 04 00:17:37.331456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:37.328561    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 04 00:17:38.331421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:38.329493    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 04 00:17:38.331421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:38.329560    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 04 00:17:39.322412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:39.321802    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 04 00:17:39.322412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:39.321868    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 04 00:17:39.471938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:39.471898    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:17:40.331837 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:40.331792    6430 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 04 00:17:40.332070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:40.331865    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 04 00:17:41.382089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:41.382049    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:17:41.383576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:41.383551    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:17:47.294054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.294019    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:17:47.304408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.301402    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.310404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.307997    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:17:47.317447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.314777    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-85ad28-serving-cert\""
Nov 04 00:17:47.348060 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.348017    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxbg\" (UniqueName: \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.348243 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.348210    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.348341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.348275    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.348341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.348323    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.449890 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.449854    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxbg\" (UniqueName: \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.450095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.449943    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.450095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.449975    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.450095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.450003    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.450696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.450666    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.452661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.452640    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.452753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.452681    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.462806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.462781    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxbg\" (UniqueName: \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\") pod \"maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.631443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.631412    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:47.989218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:47.989183    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:17:47.990431 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:17:47.990403    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c50a73_86e3_4b2a_8dae_c3d482f41059.slice/crio-42d1c22044d08bb5990970852ee7255e1ee60e0296c755f08556e0385b7a76c0 WatchSource:0}: Error finding container 42d1c22044d08bb5990970852ee7255e1ee60e0296c755f08556e0385b7a76c0: Status 404 returned error can't find the container with id 42d1c22044d08bb5990970852ee7255e1ee60e0296c755f08556e0385b7a76c0
Nov 04 00:17:48.816548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:48.816510    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" event={"ID":"a5c50a73-86e3-4b2a-8dae-c3d482f41059","Type":"ContainerStarted","Data":"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"}
Nov 04 00:17:48.817063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:48.816590    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" event={"ID":"a5c50a73-86e3-4b2a-8dae-c3d482f41059","Type":"ContainerStarted","Data":"42d1c22044d08bb5990970852ee7255e1ee60e0296c755f08556e0385b7a76c0"}
Nov 04 00:17:48.817063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:48.816638    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:17:48.838723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:48.838671    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" podStartSLOduration=1.191686921 podStartE2EDuration="1.838656998s" totalImagesPullingTime="646.970077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:48.83832943 +0000 UTC m=+2884.729391364" watchObservedRunningTime="2026-11-04 00:17:48.838656998 +0000 UTC m=+2884.729718932"
Nov 04 00:17:54.831457 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:17:54.831345    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:18:04.558798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:04.558762    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:18:04.924403 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:18:04.924282    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:18:18.923984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:18.923952    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:18:18.929266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:18.929247    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:18.950951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:18.950920    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:18:18.966370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:18.966343    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8cc1c0-openshift-default-dockercfg-nxf65\""
Nov 04 00:18:19.048900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.048860    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.048900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.048900    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049037    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049086    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049189 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049160    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049217    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049241    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4th\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049291    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.049343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.049328    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150672    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150717    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150786    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150840    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150874    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.150987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150961    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.150993    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4th\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151061    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151114    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151201    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151310    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151333    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151539    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.151900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.151868    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.153792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.153769    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.153994 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.153974    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.164664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.164617    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.164831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.164810    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4th\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.244828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.244745    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:19.553985 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.553957    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:18:19.554512 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:19.554483    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ad5939_3468_4942_86e9_1207662a873b.slice/crio-2be6278e76813ffb91fef33bc213c02abc0e95dae7642bf020a84d3f128da2f2 WatchSource:0}: Error finding container 2be6278e76813ffb91fef33bc213c02abc0e95dae7642bf020a84d3f128da2f2: Status 404 returned error can't find the container with id 2be6278e76813ffb91fef33bc213c02abc0e95dae7642bf020a84d3f128da2f2
Nov 04 00:18:19.557759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.557723    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:19.557868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.557807    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:19.557868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:19.557853    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:20.253233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:20.253193    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" event={"ID":"f4ad5939-3468-4942-86e9-1207662a873b","Type":"ContainerStarted","Data":"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"}
Nov 04 00:18:20.253643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:20.253256    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" event={"ID":"f4ad5939-3468-4942-86e9-1207662a873b","Type":"ContainerStarted","Data":"2be6278e76813ffb91fef33bc213c02abc0e95dae7642bf020a84d3f128da2f2"}
Nov 04 00:18:20.285097 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:20.285046    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podStartSLOduration=2.285032459 podStartE2EDuration="2.285032459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:20.279257365 +0000 UTC m=+2916.170319300" watchObservedRunningTime="2026-11-04 00:18:20.285032459 +0000 UTC m=+2916.176094394"
Nov 04 00:18:21.126427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.126351    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:18:21.134368 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.134339    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.139866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.139477    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:18:21.158077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.158047    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a6f7a\"/\"default-dockercfg-bz7x6\""
Nov 04 00:18:21.160363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.160336    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9a6f7a\"/\"test-model-9a6f7a-kserve-self-signed-certs\""
Nov 04 00:18:21.161572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.161546    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a6f7a\"/\"openshift-service-ca.crt\""
Nov 04 00:18:21.161821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.161804    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9a6f7a\"/\"kube-root-ca.crt\""
Nov 04 00:18:21.245246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.245217    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:21.246720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.246693    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:21.246848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.246741    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:21.286210 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286181    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7b68\" (UniqueName: \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.286688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286227    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.286688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286509    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.286688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286657    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.286688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286684    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.286868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.286726    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.388921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.388818    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.389114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.388983    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.389114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.389015    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.389114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.389061    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.389246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.389179    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7b68\" (UniqueName: \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.389246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.389210    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.391942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.391912    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.392822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.392179    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.392822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.392690    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.393014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.392929    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.399423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.395631    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.410609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.410579    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7b68\" (UniqueName: \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\") pod \"test-model-9a6f7a-kserve-65ff4466b7-2g65t\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") " pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.479872 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.479838    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:21.783650 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:21.783624    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:18:22.246098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.246062    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:22.246280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.246124    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:22.269887 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.269853    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" event={"ID":"2c87066d-c9bf-4c21-9bc3-8dd4768f483b","Type":"ContainerStarted","Data":"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"}
Nov 04 00:18:22.270057 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.269894    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" event={"ID":"2c87066d-c9bf-4c21-9bc3-8dd4768f483b","Type":"ContainerStarted","Data":"73c5a195338e138bac21ae47413b5eee61492ca35a120ba7ff686550a427f2f4"}
Nov 04 00:18:22.291066 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.291016    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" podStartSLOduration=1.291002634 podStartE2EDuration="1.291002634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:22.290143011 +0000 UTC m=+2918.181204949" watchObservedRunningTime="2026-11-04 00:18:22.291002634 +0000 UTC m=+2918.182064569"
Nov 04 00:18:22.486411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:22.485485    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:23.246068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:23.246028    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:23.246245 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:23.246099    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:24.256416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:24.252503    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:24.256416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:24.252579    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:25.253190 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:25.253138    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:25.253471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:25.253219    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:26.245283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:26.245248    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:26.245718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:26.245309    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:27.245317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:27.245285    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:27.245818 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:27.245339    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:28.245011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:28.244975    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 04 00:18:28.245213 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:28.245027    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 04 00:18:29.252170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:29.252135    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:29.252669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:29.252649    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:29.253487 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:29.253465    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:18:31.351670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.351631    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:18:31.356951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.356921    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.372381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.372350    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d94e94-serving-cert\""
Nov 04 00:18:31.376723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.376691    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:18:31.492515 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.490306    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.521464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.505349    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.521864 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.521812    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfmd\" (UniqueName: \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.522232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.522200    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.522793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.505050    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:31.572446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.572005    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:18:31.583756 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.583727    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:31.584448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.584423    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:18:31.584625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.584057    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.594027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.592700    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85ad28\"/\"default-dockercfg-wwbm4\""
Nov 04 00:18:31.594027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.593558    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85ad28\"/\"test-model-85ad28-kserve-self-signed-certs\""
Nov 04 00:18:31.594027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.593712    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85ad28\"/\"kube-root-ca.crt\""
Nov 04 00:18:31.594027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.593792    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85ad28\"/\"openshift-service-ca.crt\""
Nov 04 00:18:31.623877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.623848    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.623957    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.624061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624012    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624103    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfmd\" (UniqueName: \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.624197 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:18:31.624157    6430 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-serving-cert: secret "maas-api-e2e-inf-a-d94e94-serving-cert" not found
Nov 04 00:18:31.624197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624161    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624403 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624228    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.624403 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:18:31.624268    6430 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls podName:bae4ea80-167e-477b-a7a0-836d89dfec5a nodeName:}" failed. No retries permitted until 2026-11-04 00:18:32.124246291 +0000 UTC m=+2928.015308219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls") pod "maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" (UID: "bae4ea80-167e-477b-a7a0-836d89dfec5a") : secret "maas-api-e2e-inf-a-d94e94-serving-cert" not found
Nov 04 00:18:31.624551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624428    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dfg\" (UniqueName: \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624544    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624606    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.624732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624629    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.624899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.624878    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.627186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.627164    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.688555 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.688502    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfmd\" (UniqueName: \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:31.726259 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726224    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.726489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726315    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.726489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726406    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dfg\" (UniqueName: \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.726642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726483    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.726642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726623    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.726757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.726684    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.727455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.727187    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.727455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.727297    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.727455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.727356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.728951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.728924    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.729154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.729139    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.745980 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.745960    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dfg\" (UniqueName: \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\") pod \"test-model-85ad28-kserve-6b8dc6c5ff-5mq55\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") " pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:31.925468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:31.918751    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:32.143415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.137854    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:32.282271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.282151    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d94e94-b7456f65b-468s7\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:32.294343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.292542    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:32.307706 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.307673    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:18:32.408109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.408070    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" event={"ID":"8937062a-6e7b-4541-8e67-0890f742b453","Type":"ContainerStarted","Data":"8b33f7361b398870b6baec650a79809877f6a08ff3fb415365d875cf7075272a"}
Nov 04 00:18:32.427292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.426375    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:18:32.645630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.645597    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:18:32.648431 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:32.647753    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae4ea80_167e_477b_a7a0_836d89dfec5a.slice/crio-4f67f08704e22acac3959e01fbd1d75a4713f725449b3c0d25c69a5601ea159a WatchSource:0}: Error finding container 4f67f08704e22acac3959e01fbd1d75a4713f725449b3c0d25c69a5601ea159a: Status 404 returned error can't find the container with id 4f67f08704e22acac3959e01fbd1d75a4713f725449b3c0d25c69a5601ea159a
Nov 04 00:18:32.785124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.785088    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:18:32.792063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.792033    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.806804 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.804362    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-85ad28-openshift-default-dockercfg-5xfzh\""
Nov 04 00:18:32.808169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.807507    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:18:32.854240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854198    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854254    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854431    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854502    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854663    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854707    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854826 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854809    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854853    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nsp\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.854916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.854897    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.956858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.956763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.956858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.956814    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.956907    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.956971    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957139    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957174    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957228    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957280    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nsp\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957323    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957626    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957857 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957832    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957925 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957888    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.957925 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957908    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.958016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.957932    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.960370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.960341    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.960824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.960807    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.970426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.970362    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:32.970628 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:32.970606    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nsp\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:33.114495 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.114468    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:33.471071 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.471016    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" event={"ID":"8937062a-6e7b-4541-8e67-0890f742b453","Type":"ContainerStarted","Data":"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"}
Nov 04 00:18:33.481760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.479582    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" event={"ID":"bae4ea80-167e-477b-a7a0-836d89dfec5a","Type":"ContainerStarted","Data":"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"}
Nov 04 00:18:33.481760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.479651    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" event={"ID":"bae4ea80-167e-477b-a7a0-836d89dfec5a","Type":"ContainerStarted","Data":"4f67f08704e22acac3959e01fbd1d75a4713f725449b3c0d25c69a5601ea159a"}
Nov 04 00:18:33.481760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.479683    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:33.514704 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.509805    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" podStartSLOduration=2.509785334 podStartE2EDuration="2.509785334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:33.507620888 +0000 UTC m=+2929.398682834" watchObservedRunningTime="2026-11-04 00:18:33.509785334 +0000 UTC m=+2929.400847269"
Nov 04 00:18:33.723418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.706227    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" podStartSLOduration=2.264584042 podStartE2EDuration="2.706204365s" totalImagesPullingTime="441.620323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:33.538421814 +0000 UTC m=+2929.429483746" watchObservedRunningTime="2026-11-04 00:18:33.706204365 +0000 UTC m=+2929.597266302"
Nov 04 00:18:33.729359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.726076    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:18:33.744289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.744245    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:33.744506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.744340    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:33.744506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.744400    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:33.882361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.881791    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:18:33.887195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.887165    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.899857 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.899812    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:18:33.937960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.937920    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:33.941848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.941784    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942064 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.941936    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942148 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942099    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942195    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942320    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942470    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx67\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942572    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942625    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:33.942906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:33.942712    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044455 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044338    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044474    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044529    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044620    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044687    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044862 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044796    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044860    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx67\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044890    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044913    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.044982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.044921    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.045366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.045336    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.045531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.045507    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.045650 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.045624    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.045948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.045926    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.047092 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.047069    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.047582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.047563    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.061033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.061005    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.061195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.061047    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx67\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.238493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.238459    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:34.488263 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.488222    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" event={"ID":"41f5e629-bd13-4718-8a80-8c93acc775fa","Type":"ContainerStarted","Data":"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"}
Nov 04 00:18:34.488829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.488290    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" event={"ID":"41f5e629-bd13-4718-8a80-8c93acc775fa","Type":"ContainerStarted","Data":"da9aa1c5c5e8753dcb744f6a9af6cf37a7ea989d50c654baeb50b8c3611cfdf9"}
Nov 04 00:18:34.539124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.539060    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podStartSLOduration=2.539043573 podStartE2EDuration="2.539043573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:34.52315073 +0000 UTC m=+2930.414212665" watchObservedRunningTime="2026-11-04 00:18:34.539043573 +0000 UTC m=+2930.430105508"
Nov 04 00:18:34.544120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.544095    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:18:34.546258 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:34.546224    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdb561d_8b1e_4d7a_b95c_1053128d9bde.slice/crio-080a02f967d030a3bb3ac709523a32fd53783b6dfaf7eac9eb743d03a23920a7 WatchSource:0}: Error finding container 080a02f967d030a3bb3ac709523a32fd53783b6dfaf7eac9eb743d03a23920a7: Status 404 returned error can't find the container with id 080a02f967d030a3bb3ac709523a32fd53783b6dfaf7eac9eb743d03a23920a7
Nov 04 00:18:34.550039 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.550003    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:34.550163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.550084    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:34.550163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:34.550126    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:34.866200 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:18:34.866099    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:18:35.131433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:35.131290    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:35.132234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:35.132207    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:35.132445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:35.132420    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:35.498059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:35.497955    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" event={"ID":"acdb561d-8b1e-4d7a-b95c-1053128d9bde","Type":"ContainerStarted","Data":"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"}
Nov 04 00:18:35.498059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:35.498022    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" event={"ID":"acdb561d-8b1e-4d7a-b95c-1053128d9bde","Type":"ContainerStarted","Data":"080a02f967d030a3bb3ac709523a32fd53783b6dfaf7eac9eb743d03a23920a7"}
Nov 04 00:18:36.127417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.118875    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:36.127417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.118943    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:36.247416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.246645    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:36.247416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.246711    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:36.271750 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.271711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:36.367296 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.364126    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podStartSLOduration=3.364103956 podStartE2EDuration="3.364103956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:35.544619558 +0000 UTC m=+2931.435681494" watchObservedRunningTime="2026-11-04 00:18:36.364103956 +0000 UTC m=+2932.255165892"
Nov 04 00:18:36.370636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.370605    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:18:36.378295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.378226    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.379943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.379908    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:18:36.385574 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.385545    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-954cd9\"/\"default-dockercfg-gnmvx\""
Nov 04 00:18:36.385745 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.385726    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-954cd9\"/\"kube-root-ca.crt\""
Nov 04 00:18:36.385847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.385831    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-954cd9\"/\"openshift-service-ca.crt\""
Nov 04 00:18:36.385942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.385925    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-954cd9\"/\"test-model-954cd9-kserve-self-signed-certs\""
Nov 04 00:18:36.468024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.467668    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.468024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.467909    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.468024 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.468011    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvm6\" (UniqueName: \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.468529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.468079    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.468529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.468153    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.468529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.468269    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.569811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.569775    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.569833    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.569893    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.569927    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.570034    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.570084    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvm6\" (UniqueName: \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570681 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.570658    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.570936 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.570909    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.571113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.571099    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.572684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.572661    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.573550 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.573525    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.582868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.582845    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvm6\" (UniqueName: \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\") pod \"test-model-954cd9-kserve-6b5bc8df67-sgxbv\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") " pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:36.713471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:36.705799    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:37.120201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.120162    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:37.164499 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.163819    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:37.167664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.167633    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:18:37.264411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.255710    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:37.264411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.255772    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:37.575866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.575820    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" event={"ID":"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05","Type":"ContainerStarted","Data":"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"}
Nov 04 00:18:37.576362 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.575872    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" event={"ID":"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05","Type":"ContainerStarted","Data":"9d8ed50afe593e9e73669710c0eb09e8c6cb936c441811456c9b9ab7af2f9ae4"}
Nov 04 00:18:37.667416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.663418    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" podStartSLOduration=1.663361678 podStartE2EDuration="1.663361678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:37.647709702 +0000 UTC m=+2933.538771686" watchObservedRunningTime="2026-11-04 00:18:37.663361678 +0000 UTC m=+2933.554423615"
Nov 04 00:18:37.709870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:37.709799    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:38.127421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:38.115024    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:38.127421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:38.115092    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:38.253421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:38.253362    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:38.253718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:38.253686    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:39.129418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:39.127862    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:39.129418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:39.127948    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:39.240413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:39.239762    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:39.240413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:39.239828    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:39.494622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:39.494542    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:18:40.125412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:40.122809    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 04 00:18:40.125412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:40.122875    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 04 00:18:40.251634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:40.251002    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:40.251634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:40.251068    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:41.131667 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.131632    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:41.132050 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.132023    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:41.133078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.133055    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:18:41.243011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.238802    6430 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Nov 04 00:18:41.243011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.238871    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Nov 04 00:18:41.919993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.919950    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:41.932984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:41.932961    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:42.244720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:42.244643    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:42.245003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:42.244979    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:42.245774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:42.245752    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:18:42.666773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:42.666741    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:18:46.706827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.706784    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:46.720358 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.720329    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:46.736590 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.736556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:18:46.749921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.749870    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:18:46.756600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.756556    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.764794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.764761    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:18:46.768167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.768091    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7cbc3\"/\"default-dockercfg-vfm86\""
Nov 04 00:18:46.768506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.768230    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7cbc3\"/\"kube-root-ca.crt\""
Nov 04 00:18:46.768593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.768535    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7cbc3\"/\"test-model-a7cbc3-kserve-self-signed-certs\""
Nov 04 00:18:46.768709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.768683    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7cbc3\"/\"openshift-service-ca.crt\""
Nov 04 00:18:46.861296 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861253    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stplw\" (UniqueName: \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.861531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861437    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.861595 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861555    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.861760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861742    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.861884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861872    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.861929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.861900    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.962941    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963021    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963153    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963256    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963283    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963297    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963337    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stplw\" (UniqueName: \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.963647 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.963604    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.965594 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.965567    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.965829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.965812    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:46.979828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:46.979789    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stplw\" (UniqueName: \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\") pod \"test-model-a7cbc3-kserve-678dbc6d8f-zjrmn\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") " pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:47.086411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.077296    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:47.503440 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:47.503375    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a58384_5a7c_41a6_8693_682ea644495e.slice/crio-d13f0f8e5c6ceffc0020c7fd2a2af5dd3c3d6b057ba0637a38d3089bc8204cc3 WatchSource:0}: Error finding container d13f0f8e5c6ceffc0020c7fd2a2af5dd3c3d6b057ba0637a38d3089bc8204cc3: Status 404 returned error can't find the container with id d13f0f8e5c6ceffc0020c7fd2a2af5dd3c3d6b057ba0637a38d3089bc8204cc3
Nov 04 00:18:47.503688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.503661    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:18:47.734232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.734194    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" event={"ID":"56a58384-5a7c-41a6-8693-682ea644495e","Type":"ContainerStarted","Data":"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"}
Nov 04 00:18:47.734674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.734244    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" event={"ID":"56a58384-5a7c-41a6-8693-682ea644495e","Type":"ContainerStarted","Data":"d13f0f8e5c6ceffc0020c7fd2a2af5dd3c3d6b057ba0637a38d3089bc8204cc3"}
Nov 04 00:18:47.759116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.759049    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" podStartSLOduration=1.759031749 podStartE2EDuration="1.759031749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:47.75683492 +0000 UTC m=+2943.647896882" watchObservedRunningTime="2026-11-04 00:18:47.759031749 +0000 UTC m=+2943.650093741"
Nov 04 00:18:47.795084 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.794998    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:18:47.800048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.800006    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.813480 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.813439    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:18:47.830760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.830723    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:18:47.843218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.838851    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.856088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.856053    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:18:47.931887 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.931846    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.931887 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.931888    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.931922    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.931989    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932074    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl48\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932306 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932241    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932322    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932466 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932349    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932552    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932592    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw8p\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.932727 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932795 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932774    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.932975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932954    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.933023 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.932986    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:47.933023 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.933018    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:47.933137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:47.933064    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.034765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.034700    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.034765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.034766    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw8p\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.034818    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.034875    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.034966    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035053    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035092    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035154    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035209    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035235    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035276    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035300    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035333    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl48\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035378    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035549    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035648    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.035729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.035671    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.036428 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.036319    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.046059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.036701    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.048248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.048207    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.049482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.049451    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.054033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.052219    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.054033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.052534    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.054033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.053205    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.055200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.055172    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.057081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.057051    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.057258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.057227    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.057686 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.057660    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw8p\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.057916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.057899    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.059171 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.059044    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.059681 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.059655    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.060299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.060234    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.063242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.063207    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.067732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.067703    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl48\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.068651 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.068619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.081679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.081636    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:48.141933 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.141899    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:48.165854 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.165810    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:48.443895 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:48.443864    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c5ed76_4579_4d55_88e6_d62c15b6c1fe.slice/crio-087eb97c270a9de3e50a43e9e8edf7d51ce54ffa00ce385a5b030f920c99638a WatchSource:0}: Error finding container 087eb97c270a9de3e50a43e9e8edf7d51ce54ffa00ce385a5b030f920c99638a: Status 404 returned error can't find the container with id 087eb97c270a9de3e50a43e9e8edf7d51ce54ffa00ce385a5b030f920c99638a
Nov 04 00:18:48.444442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.444419    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:18:48.447185 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.447150    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.447295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.447252    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.447358 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.447294    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.480090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.480063    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:18:48.484054 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:18:48.484004    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffaa7ef3_10d9_4370_a4ac_b833cdf19e56.slice/crio-15797c32550802231a8a736135974c3ac3989d4945c5f884c67ecd7f8538957d WatchSource:0}: Error finding container 15797c32550802231a8a736135974c3ac3989d4945c5f884c67ecd7f8538957d: Status 404 returned error can't find the container with id 15797c32550802231a8a736135974c3ac3989d4945c5f884c67ecd7f8538957d
Nov 04 00:18:48.487628 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.487594    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.487704 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.487668    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.487741 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.487704    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:18:48.745046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.744950    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" event={"ID":"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe","Type":"ContainerStarted","Data":"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"}
Nov 04 00:18:48.745559 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.745051    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" event={"ID":"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe","Type":"ContainerStarted","Data":"087eb97c270a9de3e50a43e9e8edf7d51ce54ffa00ce385a5b030f920c99638a"}
Nov 04 00:18:48.747371 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.747332    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" event={"ID":"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56","Type":"ContainerStarted","Data":"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"}
Nov 04 00:18:48.747521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.747381    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" event={"ID":"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56","Type":"ContainerStarted","Data":"15797c32550802231a8a736135974c3ac3989d4945c5f884c67ecd7f8538957d"}
Nov 04 00:18:48.776247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.776187    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podStartSLOduration=1.7761723040000001 podStartE2EDuration="1.776172304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:48.771337708 +0000 UTC m=+2944.662399644" watchObservedRunningTime="2026-11-04 00:18:48.776172304 +0000 UTC m=+2944.667234238"
Nov 04 00:18:48.798822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:48.798763    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podStartSLOduration=1.798747836 podStartE2EDuration="1.798747836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:48.797369795 +0000 UTC m=+2944.688431731" watchObservedRunningTime="2026-11-04 00:18:48.798747836 +0000 UTC m=+2944.689809771"
Nov 04 00:18:49.142408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.142360    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:49.144078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.144050    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:49.144233 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.144104    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:49.166818 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.166762    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:49.169048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.169010    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:49.169223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:49.169077    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:50.149409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:50.147218    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:50.149409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:50.147282    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:50.186540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:50.178845    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:50.186540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:50.178913    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:51.150420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:51.147608    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:51.150420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:51.147669    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:51.183420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:51.173100    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:51.183420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:51.173162    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:52.144415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:52.142603    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:52.144415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:52.142669    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:52.166408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:52.166346    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:52.166898 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:52.166411    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:53.143410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:53.143263    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:53.143410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:53.143327    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:53.171588 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:53.169847    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:53.171588 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:53.169913    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:54.148415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:54.147054    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:54.148415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:54.147119    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:54.166308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:54.166264    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:54.166551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:54.166325    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:55.142566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:55.142523    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 04 00:18:55.143046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:55.142600    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 04 00:18:55.166736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:55.166691    6430 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 04 00:18:55.166949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:55.166766    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 04 00:18:56.147355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.147324    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:56.147813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.147791    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:56.148381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.148359    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:18:56.172444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.172408    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:56.172718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.172691    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:56.173623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:56.173600    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:18:57.078426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:57.078358    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:57.091207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:57.091172    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:18:57.603538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:57.603499    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:18:57.935299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:18:57.935219    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:19:04.859893 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:19:04.859855    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:19:11.257202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.257099    6430 generic.go:356] "Generic (PLEG): container finished" podID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerID="7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695" exitCode=0
Nov 04 00:19:11.257202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.257177    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerDied","Data":"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"}
Nov 04 00:19:11.257782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.257762    6430 scope.go:122] "RemoveContainer" containerID="7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"
Nov 04 00:19:11.260027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.259994    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:19:11.260140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.260089    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:19:11.260207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:11.260141    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:19:12.292766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:12.292730    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerStarted","Data":"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"}
Nov 04 00:19:12.612411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:12.610743    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:19:12.612411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:12.610864    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:12.612411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:12.610903    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:13.611779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:13.611738    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:13.612410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:13.612350    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:14.610418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:14.609038    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:14.610418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:14.609115    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:15.631945 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:15.631898    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:15.632442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:15.631989    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:16.626113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:16.620772    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:16.626113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:16.620852    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:17.615424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:17.611600    6430 patch_prober.go:29] interesting pod/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 04 00:19:17.615424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:17.611667    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 04 00:19:18.614299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:18.614267    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:19:18.614737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:18.614711    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:19:18.615332 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:18.615308    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:19:34.833632 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:19:34.833586    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:19:45.033452 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:19:45.033414    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:20:04.834282 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:20:04.834237    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:20:05.489771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.489732    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:20:05.495295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.495268    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.506781 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.506751    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ecd46b-serving-cert\""
Nov 04 00:20:05.509338 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.509050    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:20:05.572578 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.572544    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.572783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.572730    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.572853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.572835    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.572899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.572862    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkmq\" (UniqueName: \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.686493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.686453    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.687768 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.687741    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.688448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.688425    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkmq\" (UniqueName: \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.689281 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.689251    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.690126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.687548    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.704776 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.704739    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.718063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.708401    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkmq\" (UniqueName: \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.732071 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.731636    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:05.820873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:05.820841    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:06.218893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:06.218791    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:20:06.237729 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:20:06.235322    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb0fffb_9503_4c4d_bd46_c622adf4fe98.slice/crio-39ccd37317353c4c924ffd3a63e5ed778dc3e8d2fab2e188ca2850629b05bd5e WatchSource:0}: Error finding container 39ccd37317353c4c924ffd3a63e5ed778dc3e8d2fab2e188ca2850629b05bd5e: Status 404 returned error can't find the container with id 39ccd37317353c4c924ffd3a63e5ed778dc3e8d2fab2e188ca2850629b05bd5e
Nov 04 00:20:07.182582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:07.182476    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" event={"ID":"1cb0fffb-9503-4c4d-bd46-c622adf4fe98","Type":"ContainerStarted","Data":"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"}
Nov 04 00:20:07.182582 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:07.182544    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" event={"ID":"1cb0fffb-9503-4c4d-bd46-c622adf4fe98","Type":"ContainerStarted","Data":"39ccd37317353c4c924ffd3a63e5ed778dc3e8d2fab2e188ca2850629b05bd5e"}
Nov 04 00:20:07.182839 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:07.182648    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:07.205812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:07.205758    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" podStartSLOduration=1.637182229 podStartE2EDuration="2.205744644s" totalImagesPullingTime="568.562415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:07.205425777 +0000 UTC m=+3023.096487715" watchObservedRunningTime="2026-11-04 00:20:07.205744644 +0000 UTC m=+3023.096806580"
Nov 04 00:20:13.196721 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:13.196689    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:20:15.189743 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.189705    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:20:15.195132 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.195102    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.204235 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.204211    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d94e94\"/\"openshift-service-ca.crt\""
Nov 04 00:20:15.204693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.204670    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:20:15.205454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.205316    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d94e94\"/\"test-model-d94e94-kserve-self-signed-certs\""
Nov 04 00:20:15.205454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.205376    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d94e94\"/\"kube-root-ca.crt\""
Nov 04 00:20:15.205850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.205827    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d94e94\"/\"default-dockercfg-mpxk8\""
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.361963    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.362090    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.362120    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.362360    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.362432    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6wp\" (UniqueName: \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.364409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.362568    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.466291 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.466188    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.468643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.468613    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.469263 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.469243    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.469807 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.469789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.470506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.469741    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.471336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.466968    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.471548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.470706    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.471707 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.471688    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6wp\" (UniqueName: \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.472283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.470971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.485113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.485072    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.485379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.485357    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.509150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.509113    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6wp\" (UniqueName: \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\") pod \"test-model-d94e94-kserve-7977484df-j5nls\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") " pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.523958 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.514977    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:15.877975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:15.877919    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:20:15.878178 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:20:15.878157    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1b584f_1185_479e_b299_4017e4781746.slice/crio-b6ee81b3d51508637c5f4c2eba808b290ceafefde93a500cd6ae5ffd36642253 WatchSource:0}: Error finding container b6ee81b3d51508637c5f4c2eba808b290ceafefde93a500cd6ae5ffd36642253: Status 404 returned error can't find the container with id b6ee81b3d51508637c5f4c2eba808b290ceafefde93a500cd6ae5ffd36642253
Nov 04 00:20:16.300211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:16.300174    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" event={"ID":"9a1b584f-1185-479e-b299-4017e4781746","Type":"ContainerStarted","Data":"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"}
Nov 04 00:20:16.300660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:16.300225    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" event={"ID":"9a1b584f-1185-479e-b299-4017e4781746","Type":"ContainerStarted","Data":"b6ee81b3d51508637c5f4c2eba808b290ceafefde93a500cd6ae5ffd36642253"}
Nov 04 00:20:16.328653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:16.328599    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" podStartSLOduration=1.328584752 podStartE2EDuration="1.328584752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:16.322351422 +0000 UTC m=+3032.213413378" watchObservedRunningTime="2026-11-04 00:20:16.328584752 +0000 UTC m=+3032.219646691"
Nov 04 00:20:16.522761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:16.522731    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:25.522971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:25.522934    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:25.541476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:25.541442    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:26.393984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:26.393950    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:20:27.871829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.871792    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:20:27.877676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.877651    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.886311 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.886281    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:20:27.892149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.892118    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-757a7d\"/\"default-dockercfg-dx24h\""
Nov 04 00:20:27.892315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.892188    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-757a7d\"/\"test-model-757a7d-kserve-self-signed-certs\""
Nov 04 00:20:27.893711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.893572    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-757a7d\"/\"openshift-service-ca.crt\""
Nov 04 00:20:27.893711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.893661    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-757a7d\"/\"kube-root-ca.crt\""
Nov 04 00:20:27.949505 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949465    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfp96\" (UniqueName: \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.949665 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949530    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.949715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949687    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.949765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949728    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.949765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949747    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:27.949885 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:27.949865    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.082986    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083053    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083128    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083574    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp96\" (UniqueName: \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083629    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.083913 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.083888    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.087410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.084419    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.087410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.084553    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.101416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.095520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.121416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.114758    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.131137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.130872    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp96\" (UniqueName: \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\") pod \"test-model-757a7d-kserve-65c4576b77-qsg75\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") " pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.202443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.199704    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:28.568767 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:20:28.568731    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod178ad2bf_aad6_4ab5_a96f_53aedf955451.slice/crio-ab64caa40829f722a0d593fb3431b8797290d53e4b8e94c05b47d558ef2ec630 WatchSource:0}: Error finding container ab64caa40829f722a0d593fb3431b8797290d53e4b8e94c05b47d558ef2ec630: Status 404 returned error can't find the container with id ab64caa40829f722a0d593fb3431b8797290d53e4b8e94c05b47d558ef2ec630
Nov 04 00:20:28.570754 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:28.570727    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:20:29.410345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:29.410304    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" event={"ID":"178ad2bf-aad6-4ab5-a96f-53aedf955451","Type":"ContainerStarted","Data":"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"}
Nov 04 00:20:29.410791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:29.410360    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" event={"ID":"178ad2bf-aad6-4ab5-a96f-53aedf955451","Type":"ContainerStarted","Data":"ab64caa40829f722a0d593fb3431b8797290d53e4b8e94c05b47d558ef2ec630"}
Nov 04 00:20:29.433549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:29.433501    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" podStartSLOduration=2.433487057 podStartE2EDuration="2.433487057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:29.430624042 +0000 UTC m=+3045.321685977" watchObservedRunningTime="2026-11-04 00:20:29.433487057 +0000 UTC m=+3045.324548995"
Nov 04 00:20:30.201576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:30.201538    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:34.856331 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:20:34.856291    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:20:38.200293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:38.200255    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:38.220163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:38.220130    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:38.674935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:38.674900    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:20:56.889472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:20:56.889365    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:21:04.835905 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:21:04.835851    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:21:34.834638 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:21:34.834597    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:21:42.189490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.189429    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 04 00:21:42.190076 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.189694    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy" containerID="cri-o://54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8" gracePeriod=30
Nov 04 00:21:42.202762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.202724    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 04 00:21:42.203653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.203035    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy" containerID="cri-o://e22812dcd597ee3c770d2489264d551fa618aec182499b03c9f8a084ff49ab5e" gracePeriod=30
Nov 04 00:21:42.284754 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.284717    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 04 00:21:42.284986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:42.284935    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy" containerID="cri-o://dfb0101c8e917b14dcd485629455fb7583c90a25df5c8c3f0100ec7b922bc41f" gracePeriod=30
Nov 04 00:21:47.557380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.557174    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647196    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647249    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647417    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647479    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647538    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647828    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647884    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.647919    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.648041    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\") pod \"0d46e516-002a-49cb-87b0-c04311b0f809\" (UID: \"0d46e516-002a-49cb-87b0-c04311b0f809\") "
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.648812    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.649042    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d46e516-002a-49cb-87b0-c04311b0f809-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.649328    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.651313    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.655409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.653435    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.656310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.655978    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.671651 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.671519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.674576    6430 generic.go:356] "Generic (PLEG): container finished" podID="0d46e516-002a-49cb-87b0-c04311b0f809" containerID="54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8" exitCode=0
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.674770    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.675146    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerDied","Data":"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"}
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.675259    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5" event={"ID":"0d46e516-002a-49cb-87b0-c04311b0f809","Type":"ContainerDied","Data":"6dbc5d315d01cc9ea135e61a30980cb3d525e0ffc53051d96fea32dc50e0de46"}
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.675291    6430 scope.go:122] "RemoveContainer" containerID="54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"
Nov 04 00:21:47.676492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.675775    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "kube-api-access-9mww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.678791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.678759    6430 generic.go:356] "Generic (PLEG): container finished" podID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerID="e22812dcd597ee3c770d2489264d551fa618aec182499b03c9f8a084ff49ab5e" exitCode=0
Nov 04 00:21:47.678983 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.678959    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" event={"ID":"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66","Type":"ContainerDied","Data":"e22812dcd597ee3c770d2489264d551fa618aec182499b03c9f8a084ff49ab5e"}
Nov 04 00:21:47.685427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.681857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.685427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.681918    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token" pod "0d46e516-002a-49cb-87b0-c04311b0f809" (UID: "0d46e516-002a-49cb-87b0-c04311b0f809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.693456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.693236    6430 generic.go:356] "Generic (PLEG): container finished" podID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerID="dfb0101c8e917b14dcd485629455fb7583c90a25df5c8c3f0100ec7b922bc41f" exitCode=0
Nov 04 00:21:47.693456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.693307    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerDied","Data":"dfb0101c8e917b14dcd485629455fb7583c90a25df5c8c3f0100ec7b922bc41f"}
Nov 04 00:21:47.704911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.704877    6430 scope.go:122] "RemoveContainer" containerID="a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"
Nov 04 00:21:47.724468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.721828    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 04 00:21:47.725374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.725353    6430 scope.go:122] "RemoveContainer" containerID="54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:21:47.726216    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8\": container with ID starting with 54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8 not found: ID does not exist" containerID="54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.726274    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8"} err="failed to get container status \"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8\": rpc error: code = NotFound desc = could not find container \"54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8\": container with ID starting with 54b87ae4b537c0fc0a9a7e0ce577d61bd0110ce7e63848f556d3b62acb5806a8 not found: ID does not exist"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.726302    6430 scope.go:122] "RemoveContainer" containerID="a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:21:47.726808    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c\": container with ID starting with a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c not found: ID does not exist" containerID="a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.726847    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c"} err="failed to get container status \"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c\": rpc error: code = NotFound desc = could not find container \"a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c\": container with ID starting with a5118b190b4b4dfe95a2ae1a52a9531d50cebb867904a0895814341a26eecb4c not found: ID does not exist"
Nov 04 00:21:47.727812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.726869    6430 scope.go:122] "RemoveContainer" containerID="266c03d8800ce3d9c0f88022f87141afa4e18ec23044ab9cbec79bf6f5a531db"
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754891    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754928    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d46e516-002a-49cb-87b0-c04311b0f809-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754944    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754960    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mww7\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-kube-api-access-9mww7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754977    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d46e516-002a-49cb-87b0-c04311b0f809-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.754992    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.755005    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.761840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.755018    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d46e516-002a-49cb-87b0-c04311b0f809-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.781929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.781896    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 04 00:21:47.856365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856310    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856509    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856583    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856635    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856715    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856785    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856810    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856834    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.856888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856863    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856904    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.856934    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857006    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857060    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857105    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857137    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857137    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857162    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857237    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857307    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857162    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.857363 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857325    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857453    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857543    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\") pod \"5d029757-57ec-4103-8a27-f7ab436a2c3b\" (UID: \"5d029757-57ec-4103-8a27-f7ab436a2c3b\") "
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857587    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857504    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857590    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857620    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\") pod \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\" (UID: \"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66\") "
Nov 04 00:21:47.858046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.857906    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858607    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858631    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858646    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858660    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858674    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858689    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858702    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d029757-57ec-4103-8a27-f7ab436a2c3b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858716    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858729    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.858977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.858743    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.861778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.861575    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.861778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.861680    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "kube-api-access-4jvkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.861778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.861737    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.862141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.862116    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.863415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.863345    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.863642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.863616    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy" pod "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" (UID: "aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:21:47.863642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.863628    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:21:47.863814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.863646    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q" pod "5d029757-57ec-4103-8a27-f7ab436a2c3b" (UID: "5d029757-57ec-4103-8a27-f7ab436a2c3b"). InnerVolumeSpecName "kube-api-access-4nj6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:21:47.960506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960466    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nj6q\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-kube-api-access-4nj6q\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960504    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960517    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960530    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960543    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvkm\" (UniqueName: \"kubernetes.io/projected/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-kube-api-access-4jvkm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960557    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960568    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:47.960753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:47.960580    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d029757-57ec-4103-8a27-f7ab436a2c3b-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:21:48.003501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.003464    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 04 00:21:48.009780 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.009748    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-lkbc5"]
Nov 04 00:21:48.287992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.287956    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" path="/var/lib/kubelet/pods/0d46e516-002a-49cb-87b0-c04311b0f809/volumes"
Nov 04 00:21:48.703468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.703435    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"
Nov 04 00:21:48.703468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.703462    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl" event={"ID":"5d029757-57ec-4103-8a27-f7ab436a2c3b","Type":"ContainerDied","Data":"4981e0bb0cdcec40eb467c310ae423a64abeb778d9ee702f0c2b270aa165021c"}
Nov 04 00:21:48.704001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.703525    6430 scope.go:122] "RemoveContainer" containerID="dfb0101c8e917b14dcd485629455fb7583c90a25df5c8c3f0100ec7b922bc41f"
Nov 04 00:21:48.706523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.706497    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt" event={"ID":"aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66","Type":"ContainerDied","Data":"8ee560fcec782485924015e23b6a22dee0eaec88751d068bc0b6a5bd39ee5d27"}
Nov 04 00:21:48.706666 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.706537    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"
Nov 04 00:21:48.715924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.715900    6430 scope.go:122] "RemoveContainer" containerID="e22812dcd597ee3c770d2489264d551fa618aec182499b03c9f8a084ff49ab5e"
Nov 04 00:21:48.731415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.731372    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 04 00:21:48.737458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.737427    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-7pgtl"]
Nov 04 00:21:48.751094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.751063    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 04 00:21:48.755188 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:48.755159    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-hnctt"]
Nov 04 00:21:50.233907 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:50.233872    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" path="/var/lib/kubelet/pods/5d029757-57ec-4103-8a27-f7ab436a2c3b/volumes"
Nov 04 00:21:50.234494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:50.234475    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" path="/var/lib/kubelet/pods/aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66/volumes"
Nov 04 00:21:57.283654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:57.277587    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:21:57.283654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:21:57.278331    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy" containerID="cri-o://6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e" gracePeriod=30
Nov 04 00:22:02.578179 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.578098    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:22:02.700147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700097    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700156    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700261    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700323    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700354    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700381    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700540    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700589    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700613    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700689    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700709    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:02.700849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700728    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\") pod \"43df75bf-6403-45f4-87e6-844f1b009a06\" (UID: \"43df75bf-6403-45f4-87e6-844f1b009a06\") "
Nov 04 00:22:02.701234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.700901    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:22:02.701760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.701732    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.701861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.701764    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.701861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.701778    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.701861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.701792    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.701861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.701805    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43df75bf-6403-45f4-87e6-844f1b009a06-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.702803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.702778    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:02.702803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.702783    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:22:02.702937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.702831    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:02.702937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.702857    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr" pod "43df75bf-6403-45f4-87e6-844f1b009a06" (UID: "43df75bf-6403-45f4-87e6-844f1b009a06"). InnerVolumeSpecName "kube-api-access-ccvcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:02.803718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.803679    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43df75bf-6403-45f4-87e6-844f1b009a06-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.803718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.803718    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43df75bf-6403-45f4-87e6-844f1b009a06-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.803920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.803735    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.803920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.803749    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvcr\" (UniqueName: \"kubernetes.io/projected/43df75bf-6403-45f4-87e6-844f1b009a06-kube-api-access-ccvcr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:02.828972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.828887    6430 generic.go:356] "Generic (PLEG): container finished" podID="43df75bf-6403-45f4-87e6-844f1b009a06" containerID="6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e" exitCode=0
Nov 04 00:22:02.828972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.828943    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"
Nov 04 00:22:02.829200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.828967    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" event={"ID":"43df75bf-6403-45f4-87e6-844f1b009a06","Type":"ContainerDied","Data":"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"}
Nov 04 00:22:02.829200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.829024    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb" event={"ID":"43df75bf-6403-45f4-87e6-844f1b009a06","Type":"ContainerDied","Data":"81d2c506386fa9516a25f66a15b118497a91ab6a5c0aec1945d77e2004e14d9b"}
Nov 04 00:22:02.829200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.829047    6430 scope.go:122] "RemoveContainer" containerID="6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"
Nov 04 00:22:02.841448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.841426    6430 scope.go:122] "RemoveContainer" containerID="6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"
Nov 04 00:22:02.841747 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:02.841721    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e\": container with ID starting with 6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e not found: ID does not exist" containerID="6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"
Nov 04 00:22:02.841814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.841756    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e"} err="failed to get container status \"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e\": rpc error: code = NotFound desc = could not find container \"6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e\": container with ID starting with 6299a345abad360e483f08c3dd1a274cfb8301d768f5e802f2585fef940d547e not found: ID does not exist"
Nov 04 00:22:02.856244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.856206    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:22:02.862186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:02.862159    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xtggb"]
Nov 04 00:22:04.244661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:04.244627    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" path="/var/lib/kubelet/pods/43df75bf-6403-45f4-87e6-844f1b009a06/volumes"
Nov 04 00:22:04.834229 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:04.834187    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:22:06.186993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.186959    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:22:06.188883 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.188852    6430 cpu_manager.go:414] "containerMap: removing container" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy"
Nov 04 00:22:06.189058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189042    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy"
Nov 04 00:22:06.189290 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189272    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:06.189431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189405    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:06.189561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189468    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189477    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189497    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189505    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189580    6430 cpu_manager.go:414] "containerMap: removing container" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy"
Nov 04 00:22:06.189711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189589    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy"
Nov 04 00:22:06.189819 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189805    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa581b61-8d00-4f7d-81a2-4f2cf4b2fa66" containerName="istio-proxy"
Nov 04 00:22:06.189860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189826    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:06.189860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189840    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:06.189860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189852    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189864    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d029757-57ec-4103-8a27-f7ab436a2c3b" containerName="istio-proxy"
Nov 04 00:22:06.189954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.189881    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="43df75bf-6403-45f4-87e6-844f1b009a06" containerName="istio-proxy"
Nov 04 00:22:06.198718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.198691    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.203348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.203318    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:22:06.204429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.204367    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6382fc\"/\"kube-root-ca.crt\""
Nov 04 00:22:06.207231 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.207206    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6382fc\"/\"openshift-service-ca.crt\""
Nov 04 00:22:06.208380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.208348    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6382fc\"/\"test-model-6382fc-kserve-self-signed-certs\""
Nov 04 00:22:06.208523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.208494    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6382fc\"/\"default-dockercfg-wgvdr\""
Nov 04 00:22:06.277774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.277737    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.277950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.277911    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.278068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.278049    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2spl\" (UniqueName: \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.278107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.278099    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.278223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.278211    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.278257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.278242    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.380914    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2spl\" (UniqueName: \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.381007    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.381143    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.381183    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.381247    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.381428    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.387857    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.388491    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.394427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.389980    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.405417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.398234    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.417771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.417738    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.472557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.462886    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2spl\" (UniqueName: \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\") pod \"test-model-6382fc-kserve-c84cbddc5-b89zm\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") " pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.518446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.518410    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:06.821176 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:22:06.821141    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfabbe579_9d8a_44cf_a555_1ce9d2358d12.slice/crio-660e4eb407671b44c480d2d1850c4a930259fd0deee4fb8bc47211da09307e20 WatchSource:0}: Error finding container 660e4eb407671b44c480d2d1850c4a930259fd0deee4fb8bc47211da09307e20: Status 404 returned error can't find the container with id 660e4eb407671b44c480d2d1850c4a930259fd0deee4fb8bc47211da09307e20
Nov 04 00:22:06.821437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.821414    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:22:06.908840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.908798    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" event={"ID":"fabbe579-9d8a-44cf-a555-1ce9d2358d12","Type":"ContainerStarted","Data":"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"}
Nov 04 00:22:06.909004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.908849    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" event={"ID":"fabbe579-9d8a-44cf-a555-1ce9d2358d12","Type":"ContainerStarted","Data":"660e4eb407671b44c480d2d1850c4a930259fd0deee4fb8bc47211da09307e20"}
Nov 04 00:22:06.929803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:06.929716    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" podStartSLOduration=0.929698979 podStartE2EDuration="929.698979ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:22:06.92894866 +0000 UTC m=+3142.820010619" watchObservedRunningTime="2026-11-04 00:22:06.929698979 +0000 UTC m=+3142.820760915"
Nov 04 00:22:07.522522 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:07.522478    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:16.521510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:16.521471    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:16.534943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:16.534914    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:17.184141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:17.184112    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:22:21.091236 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:21.091201    6430 generic.go:356] "Generic (PLEG): container finished" podID="c8a4965f-af3d-4332-a17a-92947cef5216" containerID="ba8726203bf2f40165e7e2d72d284cf1a475c0ef70e08ed62532570e4cd7ed3f" exitCode=1
Nov 04 00:22:21.091753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:21.091270    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerDied","Data":"ba8726203bf2f40165e7e2d72d284cf1a475c0ef70e08ed62532570e4cd7ed3f"}
Nov 04 00:22:21.091863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:21.091845    6430 scope.go:122] "RemoveContainer" containerID="ba8726203bf2f40165e7e2d72d284cf1a475c0ef70e08ed62532570e4cd7ed3f"
Nov 04 00:22:22.102869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.102830    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerStarted","Data":"cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4"}
Nov 04 00:22:22.103383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.103066    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 04 00:22:22.105136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.105106    6430 generic.go:356] "Generic (PLEG): container finished" podID="82a246af-db2d-4e41-9c03-a36945082ffd" containerID="c98ac11da5e8df0746ec649cd8bbece9197648eb2743730a448eeca76743d564" exitCode=1
Nov 04 00:22:22.105248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.105179    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" event={"ID":"82a246af-db2d-4e41-9c03-a36945082ffd","Type":"ContainerDied","Data":"c98ac11da5e8df0746ec649cd8bbece9197648eb2743730a448eeca76743d564"}
Nov 04 00:22:22.105698 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.105682    6430 scope.go:122] "RemoveContainer" containerID="c98ac11da5e8df0746ec649cd8bbece9197648eb2743730a448eeca76743d564"
Nov 04 00:22:22.106229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:22.106214    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:22:23.115445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:23.115398    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz" event={"ID":"82a246af-db2d-4e41-9c03-a36945082ffd","Type":"ContainerStarted","Data":"4940817201a161001e0979bf83db1043dcb3c8c940f0379e71137bf5c05a7bed"}
Nov 04 00:22:23.115923 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:23.115748    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 04 00:22:26.971632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:26.971595    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:22:29.527456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.527379    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 04 00:22:29.527889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.527644    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-n76xc" podUID="099113e5-3a77-4970-9aeb-2ce66f6de53c" containerName="manager" containerID="cri-o://87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e" gracePeriod=10
Nov 04 00:22:29.907165 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.904015    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 04 00:22:29.929540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.929507    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\") pod \"099113e5-3a77-4970-9aeb-2ce66f6de53c\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") "
Nov 04 00:22:29.929868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.929855    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\" (UniqueName: \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\") pod \"099113e5-3a77-4970-9aeb-2ce66f6de53c\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") "
Nov 04 00:22:29.929959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.929949    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\") pod \"099113e5-3a77-4970-9aeb-2ce66f6de53c\" (UID: \"099113e5-3a77-4970-9aeb-2ce66f6de53c\") "
Nov 04 00:22:29.933627 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.933596    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert" pod "099113e5-3a77-4970-9aeb-2ce66f6de53c" (UID: "099113e5-3a77-4970-9aeb-2ce66f6de53c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:29.935822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.935318    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs" pod "099113e5-3a77-4970-9aeb-2ce66f6de53c" (UID: "099113e5-3a77-4970-9aeb-2ce66f6de53c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:29.936160 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:29.936135    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn" pod "099113e5-3a77-4970-9aeb-2ce66f6de53c" (UID: "099113e5-3a77-4970-9aeb-2ce66f6de53c"). InnerVolumeSpecName "kube-api-access-nrnmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:30.033122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.033088    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:30.033122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.033117    6430 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/099113e5-3a77-4970-9aeb-2ce66f6de53c-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:30.033122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.033127    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrnmn\" (UniqueName: \"kubernetes.io/projected/099113e5-3a77-4970-9aeb-2ce66f6de53c-kube-api-access-nrnmn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:30.265867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.265831    6430 generic.go:356] "Generic (PLEG): container finished" podID="099113e5-3a77-4970-9aeb-2ce66f6de53c" containerID="87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e" exitCode=0
Nov 04 00:22:30.266035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.265882    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n76xc"
Nov 04 00:22:30.266035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.265911    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n76xc" event={"ID":"099113e5-3a77-4970-9aeb-2ce66f6de53c","Type":"ContainerDied","Data":"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"}
Nov 04 00:22:30.266035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.265969    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n76xc" event={"ID":"099113e5-3a77-4970-9aeb-2ce66f6de53c","Type":"ContainerDied","Data":"99fef0ea846cc57ead56a6e67e308e5be1b497bac14099c648067523d4d68b93"}
Nov 04 00:22:30.266035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.265992    6430 scope.go:122] "RemoveContainer" containerID="87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"
Nov 04 00:22:30.284861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.284840    6430 scope.go:122] "RemoveContainer" containerID="87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"
Nov 04 00:22:30.285140 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:30.285119    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e\": container with ID starting with 87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e not found: ID does not exist" containerID="87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"
Nov 04 00:22:30.285188 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.285150    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e"} err="failed to get container status \"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e\": rpc error: code = NotFound desc = could not find container \"87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e\": container with ID starting with 87f92d5161fd3eb32c9601fe61f7f104ad3b9f86dd2397c0dc263ed43b65d58e not found: ID does not exist"
Nov 04 00:22:30.292271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.292243    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 04 00:22:30.298488 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:30.298459    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n76xc"]
Nov 04 00:22:32.236955 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:32.236911    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099113e5-3a77-4970-9aeb-2ce66f6de53c" path="/var/lib/kubelet/pods/099113e5-3a77-4970-9aeb-2ce66f6de53c/volumes"
Nov 04 00:22:33.118151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:33.118122    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 04 00:22:34.125186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:34.125147    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-lhbhz"
Nov 04 00:22:34.836158 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:34.836120    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:22:35.324077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:35.324044    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:22:35.324546 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:35.324249    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" podUID="45c09bed-5c66-460e-8deb-fc154987fa9c" containerName="maas-api" containerID="cri-o://b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c" gracePeriod=30
Nov 04 00:22:36.009498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.009472    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:22:36.192679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.192571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\" (UniqueName: \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\") pod \"45c09bed-5c66-460e-8deb-fc154987fa9c\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") "
Nov 04 00:22:36.192949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.192909    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\") pod \"45c09bed-5c66-460e-8deb-fc154987fa9c\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") "
Nov 04 00:22:36.192949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.192956    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\") pod \"45c09bed-5c66-460e-8deb-fc154987fa9c\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") "
Nov 04 00:22:36.193192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.193002    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\") pod \"45c09bed-5c66-460e-8deb-fc154987fa9c\" (UID: \"45c09bed-5c66-460e-8deb-fc154987fa9c\") "
Nov 04 00:22:36.193383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.193355    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca" pod "45c09bed-5c66-460e-8deb-fc154987fa9c" (UID: "45c09bed-5c66-460e-8deb-fc154987fa9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:22:36.194614 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.194579    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45c09bed-5c66-460e-8deb-fc154987fa9c-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:36.195337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.195315    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs" pod "45c09bed-5c66-460e-8deb-fc154987fa9c" (UID: "45c09bed-5c66-460e-8deb-fc154987fa9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:36.195618 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.195592    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b" pod "45c09bed-5c66-460e-8deb-fc154987fa9c" (UID: "45c09bed-5c66-460e-8deb-fc154987fa9c"). InnerVolumeSpecName "kube-api-access-77t7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:36.195718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.195685    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls" pod "45c09bed-5c66-460e-8deb-fc154987fa9c" (UID: "45c09bed-5c66-460e-8deb-fc154987fa9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:36.295840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.295801    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77t7b\" (UniqueName: \"kubernetes.io/projected/45c09bed-5c66-460e-8deb-fc154987fa9c-kube-api-access-77t7b\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:36.296070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.295878    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:36.296070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.295903    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45c09bed-5c66-460e-8deb-fc154987fa9c-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:36.367603 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.367551    6430 generic.go:356] "Generic (PLEG): container finished" podID="45c09bed-5c66-460e-8deb-fc154987fa9c" containerID="b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c" exitCode=0
Nov 04 00:22:36.368124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.367726    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" event={"ID":"45c09bed-5c66-460e-8deb-fc154987fa9c","Type":"ContainerDied","Data":"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"}
Nov 04 00:22:36.368124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.367781    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t" event={"ID":"45c09bed-5c66-460e-8deb-fc154987fa9c","Type":"ContainerDied","Data":"50bfcbb4bca4d4226a12c2924b8b04a0d51c091345f29ba98e2206e39a937da4"}
Nov 04 00:22:36.368124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.367810    6430 scope.go:122] "RemoveContainer" containerID="b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"
Nov 04 00:22:36.377447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.368081    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"
Nov 04 00:22:36.386575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.386549    6430 scope.go:122] "RemoveContainer" containerID="b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"
Nov 04 00:22:36.386926 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:36.386896    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c\": container with ID starting with b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c not found: ID does not exist" containerID="b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"
Nov 04 00:22:36.386979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.386937    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c"} err="failed to get container status \"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c\": rpc error: code = NotFound desc = could not find container \"b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c\": container with ID starting with b5998971864060375c1b5d36fb04526cb8ccbecf753f4ad62741163f965c2f7c not found: ID does not exist"
Nov 04 00:22:36.424686 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.424630    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:22:36.454652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:36.454584    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11301e-58d6cfddcd-szq4t"]
Nov 04 00:22:38.234229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:38.234194    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c09bed-5c66-460e-8deb-fc154987fa9c" path="/var/lib/kubelet/pods/45c09bed-5c66-460e-8deb-fc154987fa9c/volumes"
Nov 04 00:22:38.973262 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:38.973224    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:22:41.109381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.109348    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-vssxc"]
Nov 04 00:22:41.110171 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110155    6430 cpu_manager.go:414] "containerMap: removing container" podUID="099113e5-3a77-4970-9aeb-2ce66f6de53c" containerName="manager"
Nov 04 00:22:41.110226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110173    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099113e5-3a77-4970-9aeb-2ce66f6de53c" containerName="manager"
Nov 04 00:22:41.110226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110213    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:41.110226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110218    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d46e516-002a-49cb-87b0-c04311b0f809" containerName="istio-proxy"
Nov 04 00:22:41.110323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110257    6430 cpu_manager.go:414] "containerMap: removing container" podUID="45c09bed-5c66-460e-8deb-fc154987fa9c" containerName="maas-api"
Nov 04 00:22:41.110323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110264    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c09bed-5c66-460e-8deb-fc154987fa9c" containerName="maas-api"
Nov 04 00:22:41.110425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110383    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c09bed-5c66-460e-8deb-fc154987fa9c" containerName="maas-api"
Nov 04 00:22:41.110425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.110408    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="099113e5-3a77-4970-9aeb-2ce66f6de53c" containerName="manager"
Nov 04 00:22:41.115881 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.115859    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.125371 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.125340    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vssxc"]
Nov 04 00:22:41.128015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.127995    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 04 00:22:41.128127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.128104    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 04 00:22:41.128642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.128582    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 04 00:22:41.128642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.128612    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 04 00:22:41.128879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.128860    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nt628\""
Nov 04 00:22:41.201432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.201383    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-metrics-certs\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.201590 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.201565    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjf9\" (UniqueName: \"kubernetes.io/projected/2ad3722d-01d2-41f6-84f4-9e16b92c0060-kube-api-access-vsjf9\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.201666 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.201622    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-cert\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.303255 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.303223    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-metrics-certs\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.303489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.303311    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjf9\" (UniqueName: \"kubernetes.io/projected/2ad3722d-01d2-41f6-84f4-9e16b92c0060-kube-api-access-vsjf9\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.303489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.303364    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-cert\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.306591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.306569    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-cert\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.306688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.306618    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad3722d-01d2-41f6-84f4-9e16b92c0060-metrics-certs\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.316453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.316361    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjf9\" (UniqueName: \"kubernetes.io/projected/2ad3722d-01d2-41f6-84f4-9e16b92c0060-kube-api-access-vsjf9\") pod \"odh-model-controller-769875db56-vssxc\" (UID: \"2ad3722d-01d2-41f6-84f4-9e16b92c0060\") " pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.448471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.447006    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:41.949879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.949849    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vssxc"]
Nov 04 00:22:41.950695 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:22:41.950670    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad3722d_01d2_41f6_84f4_9e16b92c0060.slice/crio-4e82983ef95ab66a90ceda599657a6a22e9b196980bda1b87f5f34c69afeeeb9 WatchSource:0}: Error finding container 4e82983ef95ab66a90ceda599657a6a22e9b196980bda1b87f5f34c69afeeeb9: Status 404 returned error can't find the container with id 4e82983ef95ab66a90ceda599657a6a22e9b196980bda1b87f5f34c69afeeeb9
Nov 04 00:22:41.993922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:41.993814    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" podUID="afb79de9-ab1f-44d9-a222-2642eca7bea9" containerName="main" containerID="cri-o://8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776" gracePeriod=45
Nov 04 00:22:42.290254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.290221    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:22:42.329726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.329684    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.329894 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.329801    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\" (UniqueName: \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.329894 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.329869    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.330019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.329943    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.330019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.329996    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:42.330141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.330029    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\" (UniqueName: \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.330141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.330106    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\") pod \"afb79de9-ab1f-44d9-a222-2642eca7bea9\" (UID: \"afb79de9-ab1f-44d9-a222-2642eca7bea9\") "
Nov 04 00:22:42.330576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.330542    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:42.330576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.330557    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:42.332652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.331988    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.332652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.332015    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.332652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.332030    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.332873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.332842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:22:42.332970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.332949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "kube-api-access-s99lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:42.336834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.336812    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm" pod "afb79de9-ab1f-44d9-a222-2642eca7bea9" (UID: "afb79de9-ab1f-44d9-a222-2642eca7bea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:42.429198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.429162    6430 generic.go:356] "Generic (PLEG): container finished" podID="afb79de9-ab1f-44d9-a222-2642eca7bea9" containerID="8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776" exitCode=0
Nov 04 00:22:42.429440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.429208    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"
Nov 04 00:22:42.429440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.429245    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" event={"ID":"afb79de9-ab1f-44d9-a222-2642eca7bea9","Type":"ContainerDied","Data":"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"}
Nov 04 00:22:42.429440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.429297    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f" event={"ID":"afb79de9-ab1f-44d9-a222-2642eca7bea9","Type":"ContainerDied","Data":"b6e520ff592c0c63a15ff31ff4db5a927fee46fe6f7761a56ae465236d16052c"}
Nov 04 00:22:42.429440 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.429315    6430 scope.go:122] "RemoveContainer" containerID="8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"
Nov 04 00:22:42.436709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.436682    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb79de9-ab1f-44d9-a222-2642eca7bea9-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.436709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.436709    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb79de9-ab1f-44d9-a222-2642eca7bea9-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.436870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.436726    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99lq\" (UniqueName: \"kubernetes.io/projected/afb79de9-ab1f-44d9-a222-2642eca7bea9-kube-api-access-s99lq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:42.437573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.437538    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vssxc" event={"ID":"2ad3722d-01d2-41f6-84f4-9e16b92c0060","Type":"ContainerStarted","Data":"4e82983ef95ab66a90ceda599657a6a22e9b196980bda1b87f5f34c69afeeeb9"}
Nov 04 00:22:42.452195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.452176    6430 scope.go:122] "RemoveContainer" containerID="8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"
Nov 04 00:22:42.452612 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:42.452584    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776\": container with ID starting with 8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776 not found: ID does not exist" containerID="8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"
Nov 04 00:22:42.452708 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.452624    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776"} err="failed to get container status \"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776\": rpc error: code = NotFound desc = could not find container \"8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776\": container with ID starting with 8e424426c6bc362d91af4bb937698d5e0f9bf9191faee589bdfed58cadf1d776 not found: ID does not exist"
Nov 04 00:22:42.458937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.458913    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:22:42.462740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:42.462713    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8cc1c0/test-model-8cc1c0-kserve-5b7bc8bdc-hvl7f"]
Nov 04 00:22:43.447935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:43.447899    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vssxc" event={"ID":"2ad3722d-01d2-41f6-84f4-9e16b92c0060","Type":"ContainerStarted","Data":"2e6e18cd61e9cb043a1bfdfbd53acec56bfd3d2f2abe34ccd69427c401c2adbd"}
Nov 04 00:22:43.448677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:43.447975    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:43.471184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:43.471135    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-vssxc" podStartSLOduration=1.986737076 podStartE2EDuration="2.471122503s" totalImagesPullingTime="484.385427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:22:43.468299862 +0000 UTC m=+3179.359361796" watchObservedRunningTime="2026-11-04 00:22:43.471122503 +0000 UTC m=+3179.362184438"
Nov 04 00:22:44.234606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:44.234573    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb79de9-ab1f-44d9-a222-2642eca7bea9" path="/var/lib/kubelet/pods/afb79de9-ab1f-44d9-a222-2642eca7bea9/volumes"
Nov 04 00:22:45.460577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:45.460540    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:22:45.461093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:45.460771    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy" containerID="cri-o://c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267" gracePeriod=30
Nov 04 00:22:50.758751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.758720    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:22:50.807059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807019    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807072    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807133    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807201    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807233    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807341    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:50.807472 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807350    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807606    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807699 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807641    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:50.807762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807746    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.807904 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807880    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:50.808108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807897    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\") pod \"32980f4c-0f70-4230-b077-2d2746a25cda\" (UID: \"32980f4c-0f70-4230-b077-2d2746a25cda\") "
Nov 04 00:22:50.808271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.807900    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:22:50.808271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.808058    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:50.809106 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.809083    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.809249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.809232    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32980f4c-0f70-4230-b077-2d2746a25cda-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.809352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.809338    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.809470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.809454    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.809573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.809560    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.810343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.810312    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "kube-api-access-nxw6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:50.810489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.810357    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:22:50.810584 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.810566    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:22:50.810652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.810612    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo" pod "32980f4c-0f70-4230-b077-2d2746a25cda" (UID: "32980f4c-0f70-4230-b077-2d2746a25cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:22:50.911133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.911090    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.911133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.911123    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32980f4c-0f70-4230-b077-2d2746a25cda-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.911133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.911133    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32980f4c-0f70-4230-b077-2d2746a25cda-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:50.911378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:50.911143    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxw6x\" (UniqueName: \"kubernetes.io/projected/32980f4c-0f70-4230-b077-2d2746a25cda-kube-api-access-nxw6x\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:22:51.518914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.518879    6430 generic.go:356] "Generic (PLEG): container finished" podID="32980f4c-0f70-4230-b077-2d2746a25cda" containerID="c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267" exitCode=0
Nov 04 00:22:51.519130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.518936    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"
Nov 04 00:22:51.519130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.518960    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerDied","Data":"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"}
Nov 04 00:22:51.519130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.519021    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p" event={"ID":"32980f4c-0f70-4230-b077-2d2746a25cda","Type":"ContainerDied","Data":"9416c475365ff878d047682ae5131fd21a66156a3690bc1a5f3c16733c77368e"}
Nov 04 00:22:51.519130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.519046    6430 scope.go:122] "RemoveContainer" containerID="c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"
Nov 04 00:22:51.532575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.532552    6430 scope.go:122] "RemoveContainer" containerID="c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"
Nov 04 00:22:51.543443 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.543412    6430 scope.go:122] "RemoveContainer" containerID="c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"
Nov 04 00:22:51.543724 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:51.543704    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267\": container with ID starting with c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267 not found: ID does not exist" containerID="c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"
Nov 04 00:22:51.543783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.543733    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267"} err="failed to get container status \"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267\": rpc error: code = NotFound desc = could not find container \"c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267\": container with ID starting with c00d09d9bde5dfaab7ac8b6f6084a094e23e7a067a85487d32d2f325cdfbb267 not found: ID does not exist"
Nov 04 00:22:51.543783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.543750    6430 scope.go:122] "RemoveContainer" containerID="c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"
Nov 04 00:22:51.543985 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:51.543966    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375\": container with ID starting with c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375 not found: ID does not exist" containerID="c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"
Nov 04 00:22:51.544033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.543991    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375"} err="failed to get container status \"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375\": rpc error: code = NotFound desc = could not find container \"c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375\": container with ID starting with c576ff5ee88b55d930896869782efdb335ad79afcabd900ec74c5ac7f1b86375 not found: ID does not exist"
Nov 04 00:22:51.551483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.551456    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:22:51.555364 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:51.555341    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11301e-openshift-default-56d6b9f785-q2k7p"]
Nov 04 00:22:52.235098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:52.235058    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" path="/var/lib/kubelet/pods/32980f4c-0f70-4230-b077-2d2746a25cda/volumes"
Nov 04 00:22:54.459570 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:54.459480    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-vssxc"
Nov 04 00:22:59.651351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:59.651306    6430 generic.go:356] "Generic (PLEG): container finished" podID="c8a4965f-af3d-4332-a17a-92947cef5216" containerID="cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4" exitCode=1
Nov 04 00:22:59.651853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:59.651375    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerDied","Data":"cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4"}
Nov 04 00:22:59.651853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:59.651503    6430 scope.go:122] "RemoveContainer" containerID="ba8726203bf2f40165e7e2d72d284cf1a475c0ef70e08ed62532570e4cd7ed3f"
Nov 04 00:22:59.652010 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:22:59.651989    6430 scope.go:122] "RemoveContainer" containerID="cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4"
Nov 04 00:22:59.652652 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:22:59.652621    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-5778bb4449-rhgs4_kuadrant-system(c8a4965f-af3d-4332-a17a-92947cef5216)\"" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" podUID="c8a4965f-af3d-4332-a17a-92947cef5216"
Nov 04 00:23:03.116436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:03.116399    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 04 00:23:03.117008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:03.116955    6430 scope.go:122] "RemoveContainer" containerID="cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4"
Nov 04 00:23:03.117441 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:23:03.117378    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-5778bb4449-rhgs4_kuadrant-system(c8a4965f-af3d-4332-a17a-92947cef5216)\"" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" podUID="c8a4965f-af3d-4332-a17a-92947cef5216"
Nov 04 00:23:04.839689 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:23:04.839643    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:23:09.224522 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:09.224486    6430 scope.go:122] "RemoveContainer" containerID="cb6add4c52ec74acc6be6c83fecaa75138d8cc4a65b513f886cf292c88c515d4"
Nov 04 00:23:09.801353 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:09.801318    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4" event={"ID":"c8a4965f-af3d-4332-a17a-92947cef5216","Type":"ContainerStarted","Data":"64e1f3f4b8b97bc4841784eade4bf82a4b03b4e12355edd3cb2655a07704941a"}
Nov 04 00:23:09.801572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:09.801553    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 04 00:23:20.814526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:20.814490    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rhgs4"
Nov 04 00:23:34.877369 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:23:34.877326    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:23:35.673077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:35.673032    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:23:35.673456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:35.673360    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy" containerID="cri-o://f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968" gracePeriod=30
Nov 04 00:23:35.682407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:35.682342    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:23:35.682728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:35.682679    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy" containerID="cri-o://76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5" gracePeriod=30
Nov 04 00:23:40.988618 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:40.988588    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:23:41.027262 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.027230    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:23:41.084938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.084904    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085186    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.085431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085407    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085583 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085482    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085583 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085516    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085587    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085616    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085699    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085906 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085732    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.085992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.085968    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\") pod \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\" (UID: \"acdb561d-8b1e-4d7a-b95c-1053128d9bde\") "
Nov 04 00:23:41.086734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.086675    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:41.086840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.086735    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.086927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.086903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.087456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.087118    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.087456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.087135    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.087456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.087142    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.087456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.087179    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.087456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.087194    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.089739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.089713    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.089974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.089949    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "kube-api-access-nxx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.090063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.090039    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.096249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.096221    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo" pod "acdb561d-8b1e-4d7a-b95c-1053128d9bde" (UID: "acdb561d-8b1e-4d7a-b95c-1053128d9bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:23:41.137133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.137090    6430 generic.go:356] "Generic (PLEG): container finished" podID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerID="76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5" exitCode=0
Nov 04 00:23:41.137345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.137321    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" event={"ID":"acdb561d-8b1e-4d7a-b95c-1053128d9bde","Type":"ContainerDied","Data":"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"}
Nov 04 00:23:41.137470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.137402    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9" event={"ID":"acdb561d-8b1e-4d7a-b95c-1053128d9bde","Type":"ContainerDied","Data":"080a02f967d030a3bb3ac709523a32fd53783b6dfaf7eac9eb743d03a23920a7"}
Nov 04 00:23:41.137470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.137436    6430 scope.go:122] "RemoveContainer" containerID="76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"
Nov 04 00:23:41.137609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.137593    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"
Nov 04 00:23:41.143915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.143869    6430 generic.go:356] "Generic (PLEG): container finished" podID="f4ad5939-3468-4942-86e9-1207662a873b" containerID="f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968" exitCode=0
Nov 04 00:23:41.144086 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.144053    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" event={"ID":"f4ad5939-3468-4942-86e9-1207662a873b","Type":"ContainerDied","Data":"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"}
Nov 04 00:23:41.144164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.144100    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p" event={"ID":"f4ad5939-3468-4942-86e9-1207662a873b","Type":"ContainerDied","Data":"2be6278e76813ffb91fef33bc213c02abc0e95dae7642bf020a84d3f128da2f2"}
Nov 04 00:23:41.144226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.144178    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"
Nov 04 00:23:41.157719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.157443    6430 scope.go:122] "RemoveContainer" containerID="76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"
Nov 04 00:23:41.157864 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:23:41.157833    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5\": container with ID starting with 76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5 not found: ID does not exist" containerID="76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"
Nov 04 00:23:41.157940 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.157875    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5"} err="failed to get container status \"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5\": rpc error: code = NotFound desc = could not find container \"76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5\": container with ID starting with 76691026fc33feca1917b7ec2ad24e446bf8ef67c09b1874f9550cd85a5b4ff5 not found: ID does not exist"
Nov 04 00:23:41.157940 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.157897    6430 scope.go:122] "RemoveContainer" containerID="f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"
Nov 04 00:23:41.175331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.172994    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:23:41.177460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.177435    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-n7gw9"]
Nov 04 00:23:41.178055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.178039    6430 scope.go:122] "RemoveContainer" containerID="f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"
Nov 04 00:23:41.178381 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:23:41.178364    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968\": container with ID starting with f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968 not found: ID does not exist" containerID="f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"
Nov 04 00:23:41.178470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.178404    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968"} err="failed to get container status \"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968\": rpc error: code = NotFound desc = could not find container \"f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968\": container with ID starting with f9ecc51be029e25b459c7ab53edf1d47360f15643b5209526577cd7687b52968 not found: ID does not exist"
Nov 04 00:23:41.187880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.187853    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.187933    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188076    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188135    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188140    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.188266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188184    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188266 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188230    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188351    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.188494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188429    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188534    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.188674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188609    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188733 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188678    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\") pod \"f4ad5939-3468-4942-86e9-1207662a873b\" (UID: \"f4ad5939-3468-4942-86e9-1207662a873b\") "
Nov 04 00:23:41.188791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188732    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:23:41.188927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.188903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.189656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189628    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189661    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ad5939-3468-4942-86e9-1207662a873b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189675    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189691    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxx67\" (UniqueName: \"kubernetes.io/projected/acdb561d-8b1e-4d7a-b95c-1053128d9bde-kube-api-access-nxx67\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189704    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189718    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189732    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189746    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189760    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acdb561d-8b1e-4d7a-b95c-1053128d9bde-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.189774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.189774    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.191322 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.191298    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.191438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.191302    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:23:41.191606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.191574    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "kube-api-access-gk4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:23:41.191909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.191880    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy" pod "f4ad5939-3468-4942-86e9-1207662a873b" (UID: "f4ad5939-3468-4942-86e9-1207662a873b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:23:41.291022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.290937    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.291022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.290969    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4th\" (UniqueName: \"kubernetes.io/projected/f4ad5939-3468-4942-86e9-1207662a873b-kube-api-access-gk4th\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.291022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.290980    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ad5939-3468-4942-86e9-1207662a873b-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.291022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.290989    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ad5939-3468-4942-86e9-1207662a873b-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:23:41.476508 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.476474    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:23:41.483297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:41.483264    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-tv94p"]
Nov 04 00:23:42.234124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:42.234083    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" path="/var/lib/kubelet/pods/acdb561d-8b1e-4d7a-b95c-1053128d9bde/volumes"
Nov 04 00:23:42.234774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:42.234755    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ad5939-3468-4942-86e9-1207662a873b" path="/var/lib/kubelet/pods/f4ad5939-3468-4942-86e9-1207662a873b/volumes"
Nov 04 00:23:57.133898 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:23:57.133850    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:24:04.833258 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:04.833211    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:24:27.171869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:27.171833    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 04 00:24:27.172352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:27.172063    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" containerID="cri-o://055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07" gracePeriod=30
Nov 04 00:24:31.406715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:31.406675    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:24:31.407192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:31.406761    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:24:32.475585 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.475557    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 04 00:24:32.525120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525084    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525172    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525197    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525363    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.525535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525419    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525460    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525506    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525777 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525777 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525718    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.525777 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525742    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525809    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\") pod \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\" (UID: \"8ec0a15b-88da-482e-b10e-d95c47dc92bb\") "
Nov 04 00:24:32.525959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.525902    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.526113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.526075    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.526876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.526604    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.526876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.526631    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.526876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.526646    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.526876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.526660    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.527343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.527310    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:24:32.531972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.531929    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:32.532262 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.532207    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "kube-api-access-ppcgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:32.532361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.532331    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:24:32.532547 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.532523    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token" pod "8ec0a15b-88da-482e-b10e-d95c47dc92bb" (UID: "8ec0a15b-88da-482e-b10e-d95c47dc92bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:32.628536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.628495    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.628536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.628528    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.628536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.628543    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.628782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.628555    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec0a15b-88da-482e-b10e-d95c47dc92bb-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.628782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.628570    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcgj\" (UniqueName: \"kubernetes.io/projected/8ec0a15b-88da-482e-b10e-d95c47dc92bb-kube-api-access-ppcgj\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:32.752105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.752016    6430 generic.go:356] "Generic (PLEG): container finished" podID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerID="055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07" exitCode=0
Nov 04 00:24:32.752105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.752068    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"
Nov 04 00:24:32.752105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.752091    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerDied","Data":"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"}
Nov 04 00:24:32.752383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.752147    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85" event={"ID":"8ec0a15b-88da-482e-b10e-d95c47dc92bb","Type":"ContainerDied","Data":"b20ef148b77f1f23fe7fad9ab5faca08784c57bd97cd12c3501077dbe773c202"}
Nov 04 00:24:32.752383 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.752169    6430 scope.go:122] "RemoveContainer" containerID="055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"
Nov 04 00:24:32.764449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.764424    6430 scope.go:122] "RemoveContainer" containerID="e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"
Nov 04 00:24:32.775836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.775751    6430 scope.go:122] "RemoveContainer" containerID="055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"
Nov 04 00:24:32.776311 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:32.776285    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07\": container with ID starting with 055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07 not found: ID does not exist" containerID="055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"
Nov 04 00:24:32.776528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.776494    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07"} err="failed to get container status \"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07\": rpc error: code = NotFound desc = could not find container \"055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07\": container with ID starting with 055aa05ca2263b99baab088805424ba79e2fec9f69d38f19732ff8d4f841af07 not found: ID does not exist"
Nov 04 00:24:32.776634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.776530    6430 scope.go:122] "RemoveContainer" containerID="e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"
Nov 04 00:24:32.776860 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:32.776839    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e\": container with ID starting with e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e not found: ID does not exist" containerID="e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"
Nov 04 00:24:32.776926 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.776874    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e"} err="failed to get container status \"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e\": rpc error: code = NotFound desc = could not find container \"e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e\": container with ID starting with e3dd44f0b3a3e3873e83f5f2b8befd12a3a2ddd52430d010952a7f839aedb80e not found: ID does not exist"
Nov 04 00:24:32.780921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.780899    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 04 00:24:32.787231 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:32.787195    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ghh85"]
Nov 04 00:24:33.643694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:33.643020    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:24:34.239946 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:34.239907    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" path="/var/lib/kubelet/pods/8ec0a15b-88da-482e-b10e-d95c47dc92bb/volumes"
Nov 04 00:24:34.835709 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:34.835674    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:24:38.726624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:38.726566    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:24:48.670021 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:48.669957    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerName="main" containerID="cri-o://17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689" gracePeriod=45
Nov 04 00:24:48.726872 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:48.726813    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.207:8000/ready\": dial tcp 10.134.0.207:8000: connect: connection refused"
Nov 04 00:24:48.988577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:48.988545    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:24:49.083763 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.083634    6430 generic.go:356] "Generic (PLEG): container finished" podID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerID="17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689" exitCode=0
Nov 04 00:24:49.086935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.086899    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" event={"ID":"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05","Type":"ContainerDied","Data":"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"}
Nov 04 00:24:49.087167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.087147    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv" event={"ID":"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05","Type":"ContainerDied","Data":"9d8ed50afe593e9e73669710c0eb09e8c6cb936c441811456c9b9ab7af2f9ae4"}
Nov 04 00:24:49.087292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.087280    6430 scope.go:122] "RemoveContainer" containerID="17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"
Nov 04 00:24:49.087638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.087617    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"
Nov 04 00:24:49.104704 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.104656    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:24:49.106342 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106280    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.106522 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106365    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.106669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106614    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.106750 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106722    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.106815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106794    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.106947 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.106924    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\" (UniqueName: \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\") pod \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\" (UID: \"b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05\") "
Nov 04 00:24:49.114518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.113149    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:49.117805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.117667    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:49.117964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.117919    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:49.148430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.147217    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:49.148430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.148223    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:49.149565 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.149542    6430 scope.go:122] "RemoveContainer" containerID="17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"
Nov 04 00:24:49.152557 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:49.152528    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689\": container with ID starting with 17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689 not found: ID does not exist" containerID="17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"
Nov 04 00:24:49.152747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.152699    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689"} err="failed to get container status \"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689\": rpc error: code = NotFound desc = could not find container \"17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689\": container with ID starting with 17b28aea45c583001809a06282a7c54b912eea7a5be0413c34daa461bd0d3689 not found: ID does not exist"
Nov 04 00:24:49.154248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.154219    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6" pod "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" (UID: "b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05"). InnerVolumeSpecName "kube-api-access-nlvm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248317    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248405    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248422    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248435    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248448    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvm6\" (UniqueName: \"kubernetes.io/projected/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-kube-api-access-nlvm6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.248541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.248462    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:49.421110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.421078    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:24:49.426477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:49.426444    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-954cd9/test-model-954cd9-kserve-6b5bc8df67-sgxbv"]
Nov 04 00:24:50.234901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:50.234858    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" path="/var/lib/kubelet/pods/b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05/volumes"
Nov 04 00:24:53.751335 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:53.751276    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" podUID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" containerName="main" containerID="cri-o://6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3" gracePeriod=45
Nov 04 00:24:54.085343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.085310    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:24:54.133060 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.133024    6430 generic.go:356] "Generic (PLEG): container finished" podID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" containerID="6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3" exitCode=0
Nov 04 00:24:54.133271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.133090    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"
Nov 04 00:24:54.133271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.133122    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" event={"ID":"2c87066d-c9bf-4c21-9bc3-8dd4768f483b","Type":"ContainerDied","Data":"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"}
Nov 04 00:24:54.133271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.133212    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t" event={"ID":"2c87066d-c9bf-4c21-9bc3-8dd4768f483b","Type":"ContainerDied","Data":"73c5a195338e138bac21ae47413b5eee61492ca35a120ba7ff686550a427f2f4"}
Nov 04 00:24:54.133271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.133240    6430 scope.go:122] "RemoveContainer" containerID="6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"
Nov 04 00:24:54.148056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.148020    6430 scope.go:122] "RemoveContainer" containerID="6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"
Nov 04 00:24:54.148449 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:24:54.148426    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3\": container with ID starting with 6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3 not found: ID does not exist" containerID="6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"
Nov 04 00:24:54.148553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.148460    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3"} err="failed to get container status \"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3\": rpc error: code = NotFound desc = could not find container \"6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3\": container with ID starting with 6158102773c83430ed07feb736506ace2a52caf6e66aa0357ff3b555b7020bb3 not found: ID does not exist"
Nov 04 00:24:54.156150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156116    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156170    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156209    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\" (UniqueName: \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156259    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156503    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:54.156653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156540    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:54.156717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156650    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156695    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\") pod \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\" (UID: \"2c87066d-c9bf-4c21-9bc3-8dd4768f483b\") "
Nov 04 00:24:54.156992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.156967    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:54.157445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.157423    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.157445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.157447    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.157601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.157460    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.162430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.159579    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:24:54.162430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.159620    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:24:54.162430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.159762    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68" pod "2c87066d-c9bf-4c21-9bc3-8dd4768f483b" (UID: "2c87066d-c9bf-4c21-9bc3-8dd4768f483b"). InnerVolumeSpecName "kube-api-access-v7b68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:24:54.258759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.258707    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.258759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.258751    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7b68\" (UniqueName: \"kubernetes.io/projected/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-kube-api-access-v7b68\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.258759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.258763    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c87066d-c9bf-4c21-9bc3-8dd4768f483b-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:24:54.460416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.460272    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:24:54.474984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:54.474942    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9a6f7a/test-model-9a6f7a-kserve-65ff4466b7-2g65t"]
Nov 04 00:24:55.629713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:55.629209    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:24:56.234586 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:56.234550    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" path="/var/lib/kubelet/pods/2c87066d-c9bf-4c21-9bc3-8dd4768f483b/volumes"
Nov 04 00:24:57.154812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:57.154738    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 04 00:24:57.155324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:24:57.154972    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" containerID="cri-o://f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529" gracePeriod=30
Nov 04 00:25:00.151957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:00.151920    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:25:00.152343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:00.151999    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:25:02.529664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.529631    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 04 00:25:02.644877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.644842    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.644910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.644961    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645008    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645094    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645240    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:02.645287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645272    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:02.645422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645304    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645340    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645376    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645533    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\") pod \"67b36b6d-b234-43ce-894d-de820ad04451\" (UID: \"67b36b6d-b234-43ce-894d-de820ad04451\") "
Nov 04 00:25:02.645754 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:02.646101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.645894    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:02.646256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646228    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:02.646828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646651    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.646828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646681    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b36b6d-b234-43ce-894d-de820ad04451-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.646828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646697    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.646828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646745    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.646828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.646761    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.647954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.647926    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:02.648061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.648039    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "kube-api-access-8bdnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:02.648116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.648072    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:02.648652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.648628    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo" pod "67b36b6d-b234-43ce-894d-de820ad04451" (UID: "67b36b6d-b234-43ce-894d-de820ad04451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:25:02.748645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.748556    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdnr\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-kube-api-access-8bdnr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.748645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.748590    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b36b6d-b234-43ce-894d-de820ad04451-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.748645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.748601    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b36b6d-b234-43ce-894d-de820ad04451-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:02.748645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:02.748611    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b36b6d-b234-43ce-894d-de820ad04451-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:03.216012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.215978    6430 generic.go:356] "Generic (PLEG): container finished" podID="67b36b6d-b234-43ce-894d-de820ad04451" containerID="f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529" exitCode=0
Nov 04 00:25:03.216166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.216024    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"
Nov 04 00:25:03.216166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.216057    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" event={"ID":"67b36b6d-b234-43ce-894d-de820ad04451","Type":"ContainerDied","Data":"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"}
Nov 04 00:25:03.216166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.216112    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j" event={"ID":"67b36b6d-b234-43ce-894d-de820ad04451","Type":"ContainerDied","Data":"24e2b65f695f25d149fb6be6f9ccd09927dd640ba1ceee32e2f887319386a7dc"}
Nov 04 00:25:03.216166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.216138    6430 scope.go:122] "RemoveContainer" containerID="f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"
Nov 04 00:25:03.231437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.231374    6430 scope.go:122] "RemoveContainer" containerID="f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"
Nov 04 00:25:03.231772 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:03.231748    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529\": container with ID starting with f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529 not found: ID does not exist" containerID="f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"
Nov 04 00:25:03.231828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.231782    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529"} err="failed to get container status \"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529\": rpc error: code = NotFound desc = could not find container \"f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529\": container with ID starting with f3ba76ddf8a94ff4df96ad62521257da378f9307222517fd3f8b292758ad9529 not found: ID does not exist"
Nov 04 00:25:03.245591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.245554    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 04 00:25:03.251217 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:03.251189    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-z988j"]
Nov 04 00:25:04.173638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.173588    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" podUID="56a58384-5a7c-41a6-8693-682ea644495e" containerName="main" containerID="cri-o://9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f" gracePeriod=45
Nov 04 00:25:04.271193 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.270904    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b36b6d-b234-43ce-894d-de820ad04451" path="/var/lib/kubelet/pods/67b36b6d-b234-43ce-894d-de820ad04451/volumes"
Nov 04 00:25:04.543667 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.543638    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:25:04.600828 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.600790    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.601082 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.600905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.601223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601177    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:04.601507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601421    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:04.601507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601495    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.601646 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\" (UniqueName: \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.601712 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601674    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:04.601835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.601812    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.602575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.602553    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\") pod \"56a58384-5a7c-41a6-8693-682ea644495e\" (UID: \"56a58384-5a7c-41a6-8693-682ea644495e\") "
Nov 04 00:25:04.603655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.603627    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.603805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.603659    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.603805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.603673    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.604753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.604726    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "kube-api-access-stplw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:04.604873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.604727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:25:04.604950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.604871    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm" pod "56a58384-5a7c-41a6-8693-682ea644495e" (UID: "56a58384-5a7c-41a6-8693-682ea644495e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:04.715476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.706638    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stplw\" (UniqueName: \"kubernetes.io/projected/56a58384-5a7c-41a6-8693-682ea644495e-kube-api-access-stplw\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.715476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.706691    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a58384-5a7c-41a6-8693-682ea644495e-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.715476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:04.706707    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a58384-5a7c-41a6-8693-682ea644495e-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:04.834315 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:04.834270    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:25:05.348412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.347451    6430 generic.go:356] "Generic (PLEG): container finished" podID="56a58384-5a7c-41a6-8693-682ea644495e" containerID="9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f" exitCode=0
Nov 04 00:25:05.348412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.347576    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" event={"ID":"56a58384-5a7c-41a6-8693-682ea644495e","Type":"ContainerDied","Data":"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"}
Nov 04 00:25:05.348412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.347620    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn" event={"ID":"56a58384-5a7c-41a6-8693-682ea644495e","Type":"ContainerDied","Data":"d13f0f8e5c6ceffc0020c7fd2a2af5dd3c3d6b057ba0637a38d3089bc8204cc3"}
Nov 04 00:25:05.348412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.347655    6430 scope.go:122] "RemoveContainer" containerID="9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"
Nov 04 00:25:05.348412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.347854    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"
Nov 04 00:25:05.382592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.382559    6430 scope.go:122] "RemoveContainer" containerID="9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"
Nov 04 00:25:05.383729 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:05.383688    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f\": container with ID starting with 9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f not found: ID does not exist" containerID="9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"
Nov 04 00:25:05.383884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.383747    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f"} err="failed to get container status \"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f\": rpc error: code = NotFound desc = could not find container \"9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f\": container with ID starting with 9244985537f49c92b3028096b3f7ff33402bc2248ae6a77c75a1b2dca8f47c3f not found: ID does not exist"
Nov 04 00:25:05.404950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.404915    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:25:05.408276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:05.408136    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a7cbc3/test-model-a7cbc3-kserve-678dbc6d8f-zjrmn"]
Nov 04 00:25:06.234624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:06.234593    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a58384-5a7c-41a6-8693-682ea644495e" path="/var/lib/kubelet/pods/56a58384-5a7c-41a6-8693-682ea644495e/volumes"
Nov 04 00:25:10.651670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:10.651627    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" podUID="8937062a-6e7b-4541-8e67-0890f742b453" containerName="main" containerID="cri-o://d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518" gracePeriod=45
Nov 04 00:25:10.655420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:10.655365    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" podUID="8937062a-6e7b-4541-8e67-0890f742b453" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": dial tcp 10.134.0.204:8000: connect: connection refused"
Nov 04 00:25:10.945573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:10.945545    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:25:11.063315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063278    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\" (UniqueName: \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063422    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063453    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063507    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063667    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063705    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\" (UniqueName: \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\") pod \"8937062a-6e7b-4541-8e67-0890f742b453\" (UID: \"8937062a-6e7b-4541-8e67-0890f742b453\") "
Nov 04 00:25:11.063840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063752    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:11.063840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063821    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:11.063979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.063948    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:11.064453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.064428    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.064565 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.064456    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.064565 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.064472    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.065697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.065667    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "kube-api-access-q9dfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:11.065983 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.065951    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:11.065983 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.065962    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs" pod "8937062a-6e7b-4541-8e67-0890f742b453" (UID: "8937062a-6e7b-4541-8e67-0890f742b453"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:25:11.166506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.166408    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937062a-6e7b-4541-8e67-0890f742b453-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.166506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.166447    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dfg\" (UniqueName: \"kubernetes.io/projected/8937062a-6e7b-4541-8e67-0890f742b453-kube-api-access-q9dfg\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.166506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.166461    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937062a-6e7b-4541-8e67-0890f742b453-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:11.395031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.394998    6430 generic.go:356] "Generic (PLEG): container finished" podID="8937062a-6e7b-4541-8e67-0890f742b453" containerID="d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518" exitCode=0
Nov 04 00:25:11.395226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.395101    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" event={"ID":"8937062a-6e7b-4541-8e67-0890f742b453","Type":"ContainerDied","Data":"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"}
Nov 04 00:25:11.395226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.395137    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55" event={"ID":"8937062a-6e7b-4541-8e67-0890f742b453","Type":"ContainerDied","Data":"8b33f7361b398870b6baec650a79809877f6a08ff3fb415365d875cf7075272a"}
Nov 04 00:25:11.395226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.395159    6430 scope.go:122] "RemoveContainer" containerID="d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"
Nov 04 00:25:11.395460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.395296    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"
Nov 04 00:25:11.410619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.410596    6430 scope.go:122] "RemoveContainer" containerID="d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"
Nov 04 00:25:11.410930 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:11.410908    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518\": container with ID starting with d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518 not found: ID does not exist" containerID="d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"
Nov 04 00:25:11.410977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.410940    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518"} err="failed to get container status \"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518\": rpc error: code = NotFound desc = could not find container \"d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518\": container with ID starting with d8048a0b196c414c52bef262e9e172c36b05145f7d8b80871a2e49f77e997518 not found: ID does not exist"
Nov 04 00:25:11.423078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.423013    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:25:11.426630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:11.426606    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-85ad28/test-model-85ad28-kserve-6b8dc6c5ff-5mq55"]
Nov 04 00:25:12.121133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.121097    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:25:12.121611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.121366    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" containerID="cri-o://9f3247f1cb83977f2884ae51b806f9010f1facd722ba6608c22dd0830488fa12" gracePeriod=30
Nov 04 00:25:12.237404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.237356    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8937062a-6e7b-4541-8e67-0890f742b453" path="/var/lib/kubelet/pods/8937062a-6e7b-4541-8e67-0890f742b453/volumes"
Nov 04 00:25:12.677685 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.677649    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:25:12.678621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678594    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.678621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678617    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678632    6430 cpu_manager.go:414] "containerMap: removing container" podUID="56a58384-5a7c-41a6-8693-682ea644495e" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678640    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a58384-5a7c-41a6-8693-682ea644495e" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678665    6430 cpu_manager.go:414] "containerMap: removing container" podUID="afb79de9-ab1f-44d9-a222-2642eca7bea9" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678673    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb79de9-ab1f-44d9-a222-2642eca7bea9" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678690    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8937062a-6e7b-4541-8e67-0890f742b453" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678697    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8937062a-6e7b-4541-8e67-0890f742b453" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678717    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678724    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" containerName="main"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678733    6430 cpu_manager.go:414] "containerMap: removing container" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678740    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678754    6430 cpu_manager.go:414] "containerMap: removing container" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678763    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678774    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678780    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678792    6430 cpu_manager.go:414] "containerMap: removing container" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy"
Nov 04 00:25:12.678841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678800    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678852    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.678860    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679040    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d9af88-3fb6-4d6e-b9a7-b7c0348e6f05" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679057    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679071    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679081    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a58384-5a7c-41a6-8693-682ea644495e" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679091    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb79de9-ab1f-44d9-a222-2642eca7bea9" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679102    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8937062a-6e7b-4541-8e67-0890f742b453" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679116    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c87066d-c9bf-4c21-9bc3-8dd4768f483b" containerName="main"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679125    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdb561d-8b1e-4d7a-b95c-1053128d9bde" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679135    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679146    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679155    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ad5939-3468-4942-86e9-1207662a873b" containerName="istio-proxy"
Nov 04 00:25:12.679662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.679167    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b36b6d-b234-43ce-894d-de820ad04451" containerName="istio-proxy"
Nov 04 00:25:12.685205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.685181    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.696139 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.694835    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:25:12.698944 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.698909    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:25:12.706438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.700138    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.706438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.700164    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec0a15b-88da-482e-b10e-d95c47dc92bb" containerName="istio-proxy"
Nov 04 00:25:12.706438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.700220    6430 cpu_manager.go:414] "containerMap: removing container" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.706438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.700229    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32980f4c-0f70-4230-b077-2d2746a25cda" containerName="istio-proxy"
Nov 04 00:25:12.706438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.704869    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.739019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.736568    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:25:12.792267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792230    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.792267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792271    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.792526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792294    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6lc\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.792526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792495    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.792640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792524    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.792640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792548    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.792772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792682    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.792851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792834    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.792916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792875    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.793004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.792984    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.793062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793039    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.793107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793078    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.793157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793142    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.793211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793176    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.793277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793237    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgct5\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.793277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793267    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.793373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793359    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.793458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.793433    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895008    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895047    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895070    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgct5\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895096    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895193    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895232    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895378    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895445    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895458    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895499    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6lc\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895536    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895543    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895647    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895679    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895706    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895757    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895762    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.895900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895844    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895903    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.895946    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896033    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896043    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896102    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896135    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896707 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896491    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.896707 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896619    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.896836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.896811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.897965 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.897939    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.898198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.898173    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.898198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.898187    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.898562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.898545    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.908493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.908466    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6lc\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:12.908675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.908651    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.908760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.908745    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgct5\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\") pod \"maas-default-gateway-openshift-default-778b7b5c47-67dv2\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:12.909092 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:12.909076    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ht2tk\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:13.007234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.007152    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:13.048548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.048510    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:13.301404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.301353    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:25:13.307352 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:25:13.307322    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1903fd85_6808_4007_8cec_1aa4a137456c.slice/crio-0a46d3a16b2d41e7fa6d2b758c8e6ef94f193ec88583a6f945fb09ab6b9fe9bd WatchSource:0}: Error finding container 0a46d3a16b2d41e7fa6d2b758c8e6ef94f193ec88583a6f945fb09ab6b9fe9bd: Status 404 returned error can't find the container with id 0a46d3a16b2d41e7fa6d2b758c8e6ef94f193ec88583a6f945fb09ab6b9fe9bd
Nov 04 00:25:13.310901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.310867    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.311000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.310942    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.311000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.310970    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.415212 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.415171    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" event={"ID":"1903fd85-6808-4007-8cec-1aa4a137456c","Type":"ContainerStarted","Data":"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"}
Nov 04 00:25:13.415407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.415237    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" event={"ID":"1903fd85-6808-4007-8cec-1aa4a137456c","Type":"ContainerStarted","Data":"0a46d3a16b2d41e7fa6d2b758c8e6ef94f193ec88583a6f945fb09ab6b9fe9bd"}
Nov 04 00:25:13.433296 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.433259    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:25:13.438432 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:25:13.437514    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209517d0_0b9c_4657_8d07_1c74ab5505ed.slice/crio-a1e878353c2d16834f52a91e8d72bae4c6846e036e104218d2b198db81b499df WatchSource:0}: Error finding container a1e878353c2d16834f52a91e8d72bae4c6846e036e104218d2b198db81b499df: Status 404 returned error can't find the container with id a1e878353c2d16834f52a91e8d72bae4c6846e036e104218d2b198db81b499df
Nov 04 00:25:13.446859 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.446817    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.446963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.446914    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.447003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.446959    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:13.447003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.446942    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podStartSLOduration=1.446927222 podStartE2EDuration="1.446927222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:25:13.441848343 +0000 UTC m=+3329.332910278" watchObservedRunningTime="2026-11-04 00:25:13.446927222 +0000 UTC m=+3329.337989160"
Nov 04 00:25:13.769502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.769420    6430 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:25:13.769502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:13.769483    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:25:14.018044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.014447    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:14.022467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.018962    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:14.022467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.019026    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:14.461412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.461279    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" event={"ID":"209517d0-0b9c-4657-8d07-1c74ab5505ed","Type":"ContainerStarted","Data":"a5810fb9434db5941bc68d70b3143e47694b8651e0967c1ab7d212c398c1027a"}
Nov 04 00:25:14.461412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.461373    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" event={"ID":"209517d0-0b9c-4657-8d07-1c74ab5505ed","Type":"ContainerStarted","Data":"a1e878353c2d16834f52a91e8d72bae4c6846e036e104218d2b198db81b499df"}
Nov 04 00:25:14.490999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:14.490935    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podStartSLOduration=2.490921574 podStartE2EDuration="2.490921574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:25:14.486987194 +0000 UTC m=+3330.378049129" watchObservedRunningTime="2026-11-04 00:25:14.490921574 +0000 UTC m=+3330.381983514"
Nov 04 00:25:15.029413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:15.029130    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:15.029413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:15.029194    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:15.057412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:15.057052    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:15.059420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:15.058650    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:15.059420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:15.058707    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:16.009420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:16.008771    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:16.009420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:16.008834    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:16.054414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:16.052791    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:16.054414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:16.052851    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:17.010773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.010730    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:17.011348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.011313    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:17.056871 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.056816    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:17.057203 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.057167    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:17.580997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.579803    6430 generic.go:356] "Generic (PLEG): container finished" podID="472484d2-9979-430b-a985-dfbaf07d3d66" containerID="9f3247f1cb83977f2884ae51b806f9010f1facd722ba6608c22dd0830488fa12" exitCode=0
Nov 04 00:25:17.580997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.579869    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" event={"ID":"472484d2-9979-430b-a985-dfbaf07d3d66","Type":"ContainerDied","Data":"9f3247f1cb83977f2884ae51b806f9010f1facd722ba6608c22dd0830488fa12"}
Nov 04 00:25:17.686663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.686587    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:25:17.878592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878482    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.878592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878565    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.878843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.878843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878687    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.878843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878758    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.879012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.879012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878955    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.879111 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.878995    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.879111 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.879074    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\") pod \"472484d2-9979-430b-a985-dfbaf07d3d66\" (UID: \"472484d2-9979-430b-a985-dfbaf07d3d66\") "
Nov 04 00:25:17.879863 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.879831    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:17.880717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.880292    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:17.880717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.880458    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:17.880717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.880684    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:17.881818 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.881773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:17.882349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.882324    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:17.883377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.883348    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "kube-api-access-hrhmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:17.883782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.883692    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:17.883782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.883760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo" pod "472484d2-9979-430b-a985-dfbaf07d3d66" (UID: "472484d2-9979-430b-a985-dfbaf07d3d66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:25:17.981187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981150    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981180    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981192    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472484d2-9979-430b-a985-dfbaf07d3d66-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981201    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472484d2-9979-430b-a985-dfbaf07d3d66-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981213    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981222    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981230    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981238    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472484d2-9979-430b-a985-dfbaf07d3d66-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:17.981444 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:17.981247    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhmg\" (UniqueName: \"kubernetes.io/projected/472484d2-9979-430b-a985-dfbaf07d3d66-kube-api-access-hrhmg\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:18.046446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.042627    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:18.046446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.042696    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:18.059420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.057323    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:18.059420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.057405    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:18.596873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.594759    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx" event={"ID":"472484d2-9979-430b-a985-dfbaf07d3d66","Type":"ContainerDied","Data":"c1036251627d62a5a5a9ae3f5d9b072c47992f087db8c9e4d0c98b08431dc063"}
Nov 04 00:25:18.596873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.594818    6430 scope.go:122] "RemoveContainer" containerID="9f3247f1cb83977f2884ae51b806f9010f1facd722ba6608c22dd0830488fa12"
Nov 04 00:25:18.596873 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.594950    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"
Nov 04 00:25:18.690336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.690294    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:25:18.695439 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:18.695303    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-gx4mx"]
Nov 04 00:25:19.011480 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:19.010729    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:19.011480 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:19.010819    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:19.051411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:19.050831    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:19.051411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:19.050899    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:20.015423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:20.009773    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:20.015423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:20.009833    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:20.050901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:20.050378    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:20.050901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:20.050517    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:20.241220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:20.241178    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" path="/var/lib/kubelet/pods/472484d2-9979-430b-a985-dfbaf07d3d66/volumes"
Nov 04 00:25:21.007565 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:21.007523    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ht2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 04 00:25:21.007794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:21.007590    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 04 00:25:21.049093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:21.049044    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-67dv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 04 00:25:21.049275 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:21.049113    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 04 00:25:22.734683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.734645    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:22.736052 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.736030    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:22.737183 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.737160    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:25:22.870273 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.870239    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:22.870834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.870774    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:22.871749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:22.871727    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:25:30.088753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.088714    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:25:30.089352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.089018    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" podUID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" containerName="maas-api" containerID="cri-o://6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1" gracePeriod=30
Nov 04 00:25:30.755760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.755732    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:25:30.766058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.766026    6430 generic.go:356] "Generic (PLEG): container finished" podID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" containerID="6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1" exitCode=0
Nov 04 00:25:30.766234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.766076    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"
Nov 04 00:25:30.766234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.766118    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" event={"ID":"4d04e01a-1e9e-41d1-9ae1-48be623223e0","Type":"ContainerDied","Data":"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"}
Nov 04 00:25:30.766234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.766174    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr" event={"ID":"4d04e01a-1e9e-41d1-9ae1-48be623223e0","Type":"ContainerDied","Data":"30ff0c30bbb5a9bd47824943dac208634554ab3d876f0e5866ce2d21b2497057"}
Nov 04 00:25:30.766234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.766198    6430 scope.go:122] "RemoveContainer" containerID="6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"
Nov 04 00:25:30.776477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.776457    6430 scope.go:122] "RemoveContainer" containerID="6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"
Nov 04 00:25:30.777028 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:30.777000    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1\": container with ID starting with 6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1 not found: ID does not exist" containerID="6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"
Nov 04 00:25:30.777194 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.777167    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1"} err="failed to get container status \"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1\": rpc error: code = NotFound desc = could not find container \"6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1\": container with ID starting with 6c25d4b51366c4dc1a7bbeb6f9478df3b1ee58c5e5fdddaa9c8f43c196c246b1 not found: ID does not exist"
Nov 04 00:25:30.874110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.874069    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\" (UniqueName: \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\") pod \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") "
Nov 04 00:25:30.874312 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.874131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\") pod \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") "
Nov 04 00:25:30.874312 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.874226    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\") pod \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") "
Nov 04 00:25:30.874312 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.874286    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\") pod \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\" (UID: \"4d04e01a-1e9e-41d1-9ae1-48be623223e0\") "
Nov 04 00:25:30.874694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.874656    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca" pod "4d04e01a-1e9e-41d1-9ae1-48be623223e0" (UID: "4d04e01a-1e9e-41d1-9ae1-48be623223e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:30.875115 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.875092    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d04e01a-1e9e-41d1-9ae1-48be623223e0-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:30.876426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.876401    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs" pod "4d04e01a-1e9e-41d1-9ae1-48be623223e0" (UID: "4d04e01a-1e9e-41d1-9ae1-48be623223e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:25:30.876510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.876445    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h" pod "4d04e01a-1e9e-41d1-9ae1-48be623223e0" (UID: "4d04e01a-1e9e-41d1-9ae1-48be623223e0"). InnerVolumeSpecName "kube-api-access-8zs2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:30.876510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.876490    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls" pod "4d04e01a-1e9e-41d1-9ae1-48be623223e0" (UID: "4d04e01a-1e9e-41d1-9ae1-48be623223e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:25:30.976709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.976614    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:30.976709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.976649    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zs2h\" (UniqueName: \"kubernetes.io/projected/4d04e01a-1e9e-41d1-9ae1-48be623223e0-kube-api-access-8zs2h\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:30.976709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:30.976665    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d04e01a-1e9e-41d1-9ae1-48be623223e0-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:31.095709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:31.095677    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:25:31.098966 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:31.098940    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9a6f7a-7dc68689f5-jkndr"]
Nov 04 00:25:32.234537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:32.234500    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" path="/var/lib/kubelet/pods/4d04e01a-1e9e-41d1-9ae1-48be623223e0/volumes"
Nov 04 00:25:34.417119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:34.417079    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:25:34.832955 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:34.832921    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:25:48.985222 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:48.985186    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:25:48.985722 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:48.985430    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy" containerID="cri-o://97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2" gracePeriod=30
Nov 04 00:25:54.229943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.229906    6430 scope.go:122] "RemoveContainer" containerID="2c451b60d128f807b5d52130b903d47cf90d081b8013318ad258af2204ae99e3"
Nov 04 00:25:54.329243 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.329214    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:25:54.409954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.409919    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.409976    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410010    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410036    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410081    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410174    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410207    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410260    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410319    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:54.410662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410518    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:54.410662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410580    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\") pod \"7fe02de0-d5c4-4a25-8a66-73ea81469082\" (UID: \"7fe02de0-d5c4-4a25-8a66-73ea81469082\") "
Nov 04 00:25:54.410662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410645    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:54.411178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.410875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:25:54.411178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411150    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:54.411407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411281    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.411407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411303    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.411407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411318    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.411407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411332    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe02de0-d5c4-4a25-8a66-73ea81469082-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.411407 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.411346    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.412435 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.412383    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:25:54.412512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.412498    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:25:54.412564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.412514    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:54.412692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.412670    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp" pod "7fe02de0-d5c4-4a25-8a66-73ea81469082" (UID: "7fe02de0-d5c4-4a25-8a66-73ea81469082"). InnerVolumeSpecName "kube-api-access-s54fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:25:54.512608 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.512574    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.512608 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.512604    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.512608 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.512614    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe02de0-d5c4-4a25-8a66-73ea81469082-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.512849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.512625    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54fp\" (UniqueName: \"kubernetes.io/projected/7fe02de0-d5c4-4a25-8a66-73ea81469082-kube-api-access-s54fp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:25:54.989616 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.989576    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:25:54.989876 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:54.989846    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy" containerID="cri-o://da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde" gracePeriod=30
Nov 04 00:25:55.027346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.027311    6430 generic.go:356] "Generic (PLEG): container finished" podID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerID="97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2" exitCode=0
Nov 04 00:25:55.027524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.027353    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"
Nov 04 00:25:55.027524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.027381    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerDied","Data":"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"}
Nov 04 00:25:55.027524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.027456    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8" event={"ID":"7fe02de0-d5c4-4a25-8a66-73ea81469082","Type":"ContainerDied","Data":"149d546efea9d453a2a8c82fdb45ec36fd65e677e5a466515aa16b31038cd6ad"}
Nov 04 00:25:55.027524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.027482    6430 scope.go:122] "RemoveContainer" containerID="97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"
Nov 04 00:25:55.039836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.039820    6430 scope.go:122] "RemoveContainer" containerID="97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"
Nov 04 00:25:55.040107 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:25:55.040088    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2\": container with ID starting with 97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2 not found: ID does not exist" containerID="97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"
Nov 04 00:25:55.040163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.040116    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2"} err="failed to get container status \"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2\": rpc error: code = NotFound desc = could not find container \"97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2\": container with ID starting with 97bf3a378072c41aee8acbcc6c80ba3564182cc4f7b997d4b1e36cbc863b2ba2 not found: ID does not exist"
Nov 04 00:25:55.055306 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.055283    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:25:55.059315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:55.059294    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-k7dk8"]
Nov 04 00:25:56.233887 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:56.233853    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" path="/var/lib/kubelet/pods/7fe02de0-d5c4-4a25-8a66-73ea81469082/volumes"
Nov 04 00:25:57.798830 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.798794    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:25:57.799692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799669    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.799692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799690    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799698    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799705    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799723    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" containerName="maas-api"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799728    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" containerName="maas-api"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799757    6430 cpu_manager.go:414] "containerMap: removing container" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy"
Nov 04 00:25:57.799860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799763    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy"
Nov 04 00:25:57.800168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799895    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d04e01a-1e9e-41d1-9ae1-48be623223e0" containerName="maas-api"
Nov 04 00:25:57.800168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799911    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="472484d2-9979-430b-a985-dfbaf07d3d66" containerName="istio-proxy"
Nov 04 00:25:57.800168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799926    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.800168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.799933    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe02de0-d5c4-4a25-8a66-73ea81469082" containerName="istio-proxy"
Nov 04 00:25:57.805479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.805368    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.812667 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.812644    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-757a7d-openshift-default-dockercfg-4fwxb\""
Nov 04 00:25:57.815881 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.815856    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:25:57.892216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892182    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2kr\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892227    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892293    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892511 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892458    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892511 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892497    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892518    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892631 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892615    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892676    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.892729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.892711    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.993928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.993892    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994044    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994185    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2kr\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994223    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994244    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994257    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994382 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994361    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994500 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994482    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994531    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994587 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994549    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994601    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994685    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.994789 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.994772    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.995218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.995193    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.996691 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.996666    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:57.996942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:57.996920    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:58.007002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.006980    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:58.007251 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.007230    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2kr\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:58.124900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.124793    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:58.435815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.435781    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:25:58.436258 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:25:58.436227    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e24a1a_263c_4d7c_8737_7554bc65e822.slice/crio-31460ecf6b0e40de2e175ecd9565a96dc8b795ec8714bd8e7068c75281ba43b5 WatchSource:0}: Error finding container 31460ecf6b0e40de2e175ecd9565a96dc8b795ec8714bd8e7068c75281ba43b5: Status 404 returned error can't find the container with id 31460ecf6b0e40de2e175ecd9565a96dc8b795ec8714bd8e7068c75281ba43b5
Nov 04 00:25:58.439953 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.439920    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:58.440042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.439996    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:58.440042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:58.440029    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:25:59.060667 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.060626    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" event={"ID":"96e24a1a-263c-4d7c-8737-7554bc65e822","Type":"ContainerStarted","Data":"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"}
Nov 04 00:25:59.061453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.060696    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" event={"ID":"96e24a1a-263c-4d7c-8737-7554bc65e822","Type":"ContainerStarted","Data":"31460ecf6b0e40de2e175ecd9565a96dc8b795ec8714bd8e7068c75281ba43b5"}
Nov 04 00:25:59.096313 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.096253    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podStartSLOduration=2.09623974 podStartE2EDuration="2.09623974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:25:59.094214908 +0000 UTC m=+3374.985276845" watchObservedRunningTime="2026-11-04 00:25:59.09623974 +0000 UTC m=+3374.987301675"
Nov 04 00:25:59.126411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.125899    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:25:59.132902 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.132868    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:25:59.133083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:25:59.132926    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:00.133870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.131827    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:26:00.133870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.131892    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:00.350442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.350373    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:26:00.363612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363580    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363638    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363676    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363710    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363794    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363867    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:00.363952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363882    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.363995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.363976    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.364250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364075    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.364250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364121    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\") pod \"65a2ad27-1b90-42ff-8977-abb226bc11c3\" (UID: \"65a2ad27-1b90-42ff-8977-abb226bc11c3\") "
Nov 04 00:26:00.364250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364172    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:00.364250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364205    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:00.364536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364320    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:00.364536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364407    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:00.364867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364844    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.364867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364866    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.365033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364876    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.365033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364891    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2ad27-1b90-42ff-8977-abb226bc11c3-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.365033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.364902    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.366035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.366015    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:00.366359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.366336    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:00.366468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.366367    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "kube-api-access-d974s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:00.366468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.366428    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo" pod "65a2ad27-1b90-42ff-8977-abb226bc11c3" (UID: "65a2ad27-1b90-42ff-8977-abb226bc11c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:00.466265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.466231    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d974s\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-kube-api-access-d974s\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.466265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.466263    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.466502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.466274    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:00.466502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:00.466283    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2ad27-1b90-42ff-8977-abb226bc11c3-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:01.078468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.078429    6430 generic.go:356] "Generic (PLEG): container finished" podID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerID="da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde" exitCode=0
Nov 04 00:26:01.078687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.078479    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"
Nov 04 00:26:01.078687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.078530    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerDied","Data":"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"}
Nov 04 00:26:01.078687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.078596    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87" event={"ID":"65a2ad27-1b90-42ff-8977-abb226bc11c3","Type":"ContainerDied","Data":"a758fb632595688e5e265e40123195880030fc0efe7a2205b4a1ecd8ae19b6d6"}
Nov 04 00:26:01.078687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.078626    6430 scope.go:122] "RemoveContainer" containerID="da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"
Nov 04 00:26:01.098619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.098594    6430 scope.go:122] "RemoveContainer" containerID="7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"
Nov 04 00:26:01.132704 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.132671    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:26:01.136412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.136024    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a7cbc3-openshift-default-84898bf96c-f7p87"]
Nov 04 00:26:01.136788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.136674    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:26:01.136788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.136736    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:01.141739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.141632    6430 scope.go:122] "RemoveContainer" containerID="da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"
Nov 04 00:26:01.142048 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:01.142019    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde\": container with ID starting with da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde not found: ID does not exist" containerID="da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"
Nov 04 00:26:01.142177 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.142058    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde"} err="failed to get container status \"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde\": rpc error: code = NotFound desc = could not find container \"da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde\": container with ID starting with da0ceba1b9ea3c4c71b56042c8d9c71c15ff0bfe5796488bf0ce92d8fe5bffde not found: ID does not exist"
Nov 04 00:26:01.142177 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.142082    6430 scope.go:122] "RemoveContainer" containerID="7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"
Nov 04 00:26:01.142712 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:01.142377    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695\": container with ID starting with 7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695 not found: ID does not exist" containerID="7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"
Nov 04 00:26:01.142765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:01.142722    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695"} err="failed to get container status \"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695\": rpc error: code = NotFound desc = could not find container \"7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695\": container with ID starting with 7f6877054437b5308cdc7210b16b1719cf37b84c3f411f2b1f7313c5743f8695 not found: ID does not exist"
Nov 04 00:26:02.132623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:02.128732    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:26:02.132623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:02.128793    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:02.241536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:02.241504    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" path="/var/lib/kubelet/pods/65a2ad27-1b90-42ff-8977-abb226bc11c3/volumes"
Nov 04 00:26:03.132415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:03.132318    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:26:03.132720 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:03.132685    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:04.130427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:04.129659    6430 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 04 00:26:04.130427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:04.129723    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 04 00:26:04.834610 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:04.834561    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:26:05.131319 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:05.131233    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:26:05.131850 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:05.131692    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:26:05.132226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:05.132206    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:26:06.496142 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:06.496106    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:26:06.496963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:06.496923    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" podUID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" containerName="maas-api" containerID="cri-o://36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc" gracePeriod=30
Nov 04 00:26:06.973797 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:06.973769    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:26:07.163413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.142066    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\" (UniqueName: \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\") pod \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") "
Nov 04 00:26:07.163803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.163769    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\") pod \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") "
Nov 04 00:26:07.163996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.163977    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\") pod \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") "
Nov 04 00:26:07.164355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.164327    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\") pod \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\" (UID: \"a5c50a73-86e3-4b2a-8dae-c3d482f41059\") "
Nov 04 00:26:07.169853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.168970    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca" pod "a5c50a73-86e3-4b2a-8dae-c3d482f41059" (UID: "a5c50a73-86e3-4b2a-8dae-c3d482f41059"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:07.175217 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.175165    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs" pod "a5c50a73-86e3-4b2a-8dae-c3d482f41059" (UID: "a5c50a73-86e3-4b2a-8dae-c3d482f41059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:07.179468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.175809    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg" pod "a5c50a73-86e3-4b2a-8dae-c3d482f41059" (UID: "a5c50a73-86e3-4b2a-8dae-c3d482f41059"). InnerVolumeSpecName "kube-api-access-rbxbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:07.180764 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.180733    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls" pod "a5c50a73-86e3-4b2a-8dae-c3d482f41059" (UID: "a5c50a73-86e3-4b2a-8dae-c3d482f41059"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:07.191832 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.191805    6430 generic.go:356] "Generic (PLEG): container finished" podID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" containerID="36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc" exitCode=0
Nov 04 00:26:07.191941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.191848    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" event={"ID":"a5c50a73-86e3-4b2a-8dae-c3d482f41059","Type":"ContainerDied","Data":"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"}
Nov 04 00:26:07.191941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.191911    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx" event={"ID":"a5c50a73-86e3-4b2a-8dae-c3d482f41059","Type":"ContainerDied","Data":"42d1c22044d08bb5990970852ee7255e1ee60e0296c755f08556e0385b7a76c0"}
Nov 04 00:26:07.191941 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.191938    6430 scope.go:122] "RemoveContainer" containerID="36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"
Nov 04 00:26:07.192057 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.191863    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"
Nov 04 00:26:07.203116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.203097    6430 scope.go:122] "RemoveContainer" containerID="36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"
Nov 04 00:26:07.203484 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:07.203399    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc\": container with ID starting with 36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc not found: ID does not exist" containerID="36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"
Nov 04 00:26:07.203484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.203465    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc"} err="failed to get container status \"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc\": rpc error: code = NotFound desc = could not find container \"36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc\": container with ID starting with 36acf0ce4fc2db998f2a5bcb5a0fd590b5996320248f668090187829c28804fc not found: ID does not exist"
Nov 04 00:26:07.218967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.218936    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:26:07.222794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.222768    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85ad28-97f6b6d9d-f7tsx"]
Nov 04 00:26:07.267006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.266972    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:07.267006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.267003    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c50a73-86e3-4b2a-8dae-c3d482f41059-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:07.267190 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.267019    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxbg\" (UniqueName: \"kubernetes.io/projected/a5c50a73-86e3-4b2a-8dae-c3d482f41059-kube-api-access-rbxbg\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:07.267190 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:07.267032    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c50a73-86e3-4b2a-8dae-c3d482f41059-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:08.233688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:08.233653    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" path="/var/lib/kubelet/pods/a5c50a73-86e3-4b2a-8dae-c3d482f41059/volumes"
Nov 04 00:26:10.327697 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:10.327659    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:26:15.697481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.697434    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:26:15.697997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.697671    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy" containerID="cri-o://f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc" gracePeriod=30
Nov 04 00:26:15.701572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.701538    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:26:15.701813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.701782    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy" containerID="cri-o://4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1" gracePeriod=30
Nov 04 00:26:15.708511 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.708480    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:26:15.708672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:15.708650    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy" containerID="cri-o://95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197" gracePeriod=30
Nov 04 00:26:21.030694 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.030658    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:26:21.063314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.058604    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:26:21.065188 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065156    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065325 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065230    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065382    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065447    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065546    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065609    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065644    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065676    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.065832 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.065814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\") pod \"41f5e629-bd13-4718-8a80-8c93acc775fa\" (UID: \"41f5e629-bd13-4718-8a80-8c93acc775fa\") "
Nov 04 00:26:21.066849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.066818    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.067023 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067003    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.067552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067141    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.067552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067162    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.067552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067310    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:21.067552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067358    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.067815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.067648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.070121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.070094    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:21.071970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.071383    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "kube-api-access-x8nsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.074901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.074871    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.078367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.078343    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy" pod "41f5e629-bd13-4718-8a80-8c93acc775fa" (UID: "41f5e629-bd13-4718-8a80-8c93acc775fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.114762 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.114736    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:26:21.168881 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.168840    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.168898    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.168932    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.168967    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169034    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169072    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169140    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169308 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.169501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169421    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.169501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169436    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:21.169621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169472    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169534    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169543    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.169621 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169576    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.169831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169725    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169773    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169879    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169908    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.169989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169952    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\") pod \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\" (UID: \"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe\") "
Nov 04 00:26:21.170137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170007    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.170137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170047    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\") pod \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\" (UID: \"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56\") "
Nov 04 00:26:21.170986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170956    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nsp\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-kube-api-access-x8nsp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170988    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171004    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171017    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171029    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171042    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171057    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171070    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f5e629-bd13-4718-8a80-8c93acc775fa-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171083    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f5e629-bd13-4718-8a80-8c93acc775fa-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171097    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171111    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f5e629-bd13-4718-8a80-8c93acc775fa-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169773    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.169879    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170265    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.170628    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171075    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.171730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171257    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.172070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.171983    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:21.173557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.173530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "kube-api-access-ldl48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.173678 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.173633    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:26:21.173678 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.173665    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.174149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.174120    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token" pod "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" (UID: "e5c5ed76-4579-4d55-88e6-d62c15b6c1fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.174225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.174187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "kube-api-access-cqw8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.174728 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.174704    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:21.175155 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.175133    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy" pod "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" (UID: "ffaa7ef3-10d9-4370-a4ac-b833cdf19e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272711    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl48\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-kube-api-access-ldl48\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272741    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272755    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272769    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272781    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.272792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272794    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272806    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272819    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272831    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272842    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272855    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw8p\" (UniqueName: \"kubernetes.io/projected/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-kube-api-access-cqw8p\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272868    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272880    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.273256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.272892    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:21.349255 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.349221    6430 generic.go:356] "Generic (PLEG): container finished" podID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerID="4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1" exitCode=0
Nov 04 00:26:21.349467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.349282    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"
Nov 04 00:26:21.349467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.349302    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" event={"ID":"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe","Type":"ContainerDied","Data":"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"}
Nov 04 00:26:21.349467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.349359    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd" event={"ID":"e5c5ed76-4579-4d55-88e6-d62c15b6c1fe","Type":"ContainerDied","Data":"087eb97c270a9de3e50a43e9e8edf7d51ce54ffa00ce385a5b030f920c99638a"}
Nov 04 00:26:21.349467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.349425    6430 scope.go:122] "RemoveContainer" containerID="4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"
Nov 04 00:26:21.350905 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.350884    6430 generic.go:356] "Generic (PLEG): container finished" podID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerID="f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc" exitCode=0
Nov 04 00:26:21.351048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.350922    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"
Nov 04 00:26:21.351048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.350965    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" event={"ID":"41f5e629-bd13-4718-8a80-8c93acc775fa","Type":"ContainerDied","Data":"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"}
Nov 04 00:26:21.351149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.351055    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8" event={"ID":"41f5e629-bd13-4718-8a80-8c93acc775fa","Type":"ContainerDied","Data":"da9aa1c5c5e8753dcb744f6a9af6cf37a7ea989d50c654baeb50b8c3611cfdf9"}
Nov 04 00:26:21.352715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.352684    6430 generic.go:356] "Generic (PLEG): container finished" podID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerID="95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197" exitCode=0
Nov 04 00:26:21.352836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.352725    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"
Nov 04 00:26:21.352836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.352758    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" event={"ID":"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56","Type":"ContainerDied","Data":"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"}
Nov 04 00:26:21.352836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.352808    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls" event={"ID":"ffaa7ef3-10d9-4370-a4ac-b833cdf19e56","Type":"ContainerDied","Data":"15797c32550802231a8a736135974c3ac3989d4945c5f884c67ecd7f8538957d"}
Nov 04 00:26:21.363791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.363771    6430 scope.go:122] "RemoveContainer" containerID="4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"
Nov 04 00:26:21.364064 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:21.364038    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1\": container with ID starting with 4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1 not found: ID does not exist" containerID="4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"
Nov 04 00:26:21.364150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.364073    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1"} err="failed to get container status \"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1\": rpc error: code = NotFound desc = could not find container \"4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1\": container with ID starting with 4ac7783143022664fdc4dad787f743f6b2f905a96bc9065bb7098432229691a1 not found: ID does not exist"
Nov 04 00:26:21.364150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.364093    6430 scope.go:122] "RemoveContainer" containerID="f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"
Nov 04 00:26:21.374916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.374896    6430 scope.go:122] "RemoveContainer" containerID="f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"
Nov 04 00:26:21.375197 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:21.375175    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc\": container with ID starting with f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc not found: ID does not exist" containerID="f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"
Nov 04 00:26:21.375240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.375205    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc"} err="failed to get container status \"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc\": rpc error: code = NotFound desc = could not find container \"f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc\": container with ID starting with f35abfeebd3591c670bf9979cb3149e6e7f146c96d1095c97147a9760ca5dafc not found: ID does not exist"
Nov 04 00:26:21.375240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.375222    6430 scope.go:122] "RemoveContainer" containerID="95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"
Nov 04 00:26:21.389827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.389727    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:26:21.400404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.400364    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-wgnls"]
Nov 04 00:26:21.400548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.400528    6430 scope.go:122] "RemoveContainer" containerID="95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"
Nov 04 00:26:21.403854 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:21.403819    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197\": container with ID starting with 95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197 not found: ID does not exist" containerID="95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"
Nov 04 00:26:21.404000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.403864    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197"} err="failed to get container status \"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197\": rpc error: code = NotFound desc = could not find container \"95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197\": container with ID starting with 95ba3c28c6089b01b3de6ca1fe165617e75d1b97eb388a2325e76a332d8a6197 not found: ID does not exist"
Nov 04 00:26:21.430294 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.417124    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:26:21.435250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.435218    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-bmdgd"]
Nov 04 00:26:21.442473 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.442440    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:26:21.452419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.452375    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:26:21.458102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:21.458076    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7wsq8"]
Nov 04 00:26:22.233689 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:22.233654    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" path="/var/lib/kubelet/pods/41f5e629-bd13-4718-8a80-8c93acc775fa/volumes"
Nov 04 00:26:22.234177 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:22.234162    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" path="/var/lib/kubelet/pods/e5c5ed76-4579-4d55-88e6-d62c15b6c1fe/volumes"
Nov 04 00:26:22.234636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:22.234620    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" path="/var/lib/kubelet/pods/ffaa7ef3-10d9-4370-a4ac-b833cdf19e56/volumes"
Nov 04 00:26:25.350452 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.350410    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" podUID="9a1b584f-1185-479e-b299-4017e4781746" containerName="main" containerID="cri-o://a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619" gracePeriod=45
Nov 04 00:26:25.381593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.381555    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" podUID="9a1b584f-1185-479e-b299-4017e4781746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": dial tcp 10.134.0.212:8000: connect: connection refused"
Nov 04 00:26:25.617033 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.616965    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:26:25.758766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.758726    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.783260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.778527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.784338    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.784538    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.784722    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\" (UniqueName: \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.784870    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.784899    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\") pod \"9a1b584f-1185-479e-b299-4017e4781746\" (UID: \"9a1b584f-1185-479e-b299-4017e4781746\") "
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.788132    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:25.790025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.788412    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:25.790622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.790540    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:25.790622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.790564    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:25.790622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.790578    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:25.799730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.794555    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:25.799730 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.795609    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:25.800599 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.800487    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp" pod "9a1b584f-1185-479e-b299-4017e4781746" (UID: "9a1b584f-1185-479e-b299-4017e4781746"). InnerVolumeSpecName "kube-api-access-nm6wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:25.892492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.892407    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6wp\" (UniqueName: \"kubernetes.io/projected/9a1b584f-1185-479e-b299-4017e4781746-kube-api-access-nm6wp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:25.892492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.892441    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1b584f-1185-479e-b299-4017e4781746-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:25.892492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:25.892453    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1b584f-1185-479e-b299-4017e4781746-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:26.395555 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.395517    6430 generic.go:356] "Generic (PLEG): container finished" podID="9a1b584f-1185-479e-b299-4017e4781746" containerID="a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619" exitCode=0
Nov 04 00:26:26.396040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.395607    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" event={"ID":"9a1b584f-1185-479e-b299-4017e4781746","Type":"ContainerDied","Data":"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"}
Nov 04 00:26:26.396040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.395646    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls" event={"ID":"9a1b584f-1185-479e-b299-4017e4781746","Type":"ContainerDied","Data":"b6ee81b3d51508637c5f4c2eba808b290ceafefde93a500cd6ae5ffd36642253"}
Nov 04 00:26:26.396040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.395671    6430 scope.go:122] "RemoveContainer" containerID="a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"
Nov 04 00:26:26.396040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.395838    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"
Nov 04 00:26:26.407400 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.407369    6430 scope.go:122] "RemoveContainer" containerID="a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"
Nov 04 00:26:26.407764 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:26.407744    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619\": container with ID starting with a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619 not found: ID does not exist" containerID="a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"
Nov 04 00:26:26.407831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.407772    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619"} err="failed to get container status \"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619\": rpc error: code = NotFound desc = could not find container \"a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619\": container with ID starting with a76758e6cfb35e8c5deabce4d257abbf2176763e38c6d9e07397856c9af5b619 not found: ID does not exist"
Nov 04 00:26:26.419194 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.419166    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:26:26.422351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:26.422326    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d94e94/test-model-d94e94-kserve-7977484df-j5nls"]
Nov 04 00:26:28.232955 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:28.232920    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1b584f-1185-479e-b299-4017e4781746" path="/var/lib/kubelet/pods/9a1b584f-1185-479e-b299-4017e4781746/volumes"
Nov 04 00:26:34.832530 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:34.832487    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:26:36.470065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.470023    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerName="main" containerID="cri-o://2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667" gracePeriod=45
Nov 04 00:26:36.796002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.795974    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:26:36.864465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864423    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.864661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864612    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\" (UniqueName: \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.864661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864645    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:36.864803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864709    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.864803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864775    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.864947 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864916    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.865019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.864941    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:36.865078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.865014    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:36.865078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.865033    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\" (UniqueName: \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\") pod \"178ad2bf-aad6-4ab5-a96f-53aedf955451\" (UID: \"178ad2bf-aad6-4ab5-a96f-53aedf955451\") "
Nov 04 00:26:36.865718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.865685    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:36.865718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.865712    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:36.865879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.865727    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:36.867688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.867653    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:26:36.867813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.867683    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "kube-api-access-rfp96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:36.867813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.867692    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs" pod "178ad2bf-aad6-4ab5-a96f-53aedf955451" (UID: "178ad2bf-aad6-4ab5-a96f-53aedf955451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:36.967611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.967572    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178ad2bf-aad6-4ab5-a96f-53aedf955451-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:36.967611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.967605    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178ad2bf-aad6-4ab5-a96f-53aedf955451-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:36.967611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:36.967616    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfp96\" (UniqueName: \"kubernetes.io/projected/178ad2bf-aad6-4ab5-a96f-53aedf955451-kube-api-access-rfp96\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:37.525742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.525700    6430 generic.go:356] "Generic (PLEG): container finished" podID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerID="2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667" exitCode=0
Nov 04 00:26:37.525742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.525736    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" event={"ID":"178ad2bf-aad6-4ab5-a96f-53aedf955451","Type":"ContainerDied","Data":"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"}
Nov 04 00:26:37.526317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.525789    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" event={"ID":"178ad2bf-aad6-4ab5-a96f-53aedf955451","Type":"ContainerDied","Data":"ab64caa40829f722a0d593fb3431b8797290d53e4b8e94c05b47d558ef2ec630"}
Nov 04 00:26:37.526317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.525806    6430 scope.go:122] "RemoveContainer" containerID="2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"
Nov 04 00:26:37.526317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.525747    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"
Nov 04 00:26:37.537687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.537666    6430 scope.go:122] "RemoveContainer" containerID="2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"
Nov 04 00:26:37.537955 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:37.537935    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667\": container with ID starting with 2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667 not found: ID does not exist" containerID="2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"
Nov 04 00:26:37.538000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.537966    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667"} err="failed to get container status \"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667\": rpc error: code = NotFound desc = could not find container \"2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667\": container with ID starting with 2e2c046db2ddd507c42230f43c6731d477e4b5b0171f99d43981bbf9c022c667 not found: ID does not exist"
Nov 04 00:26:37.553330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.553289    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:26:37.556784 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.556761    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75"]
Nov 04 00:26:37.659758 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:37.659716    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-757a7d/test-model-757a7d-kserve-65c4576b77-qsg75" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:26:38.234419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:38.234368    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" path="/var/lib/kubelet/pods/178ad2bf-aad6-4ab5-a96f-53aedf955451/volumes"
Nov 04 00:26:48.256874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:48.256835    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:26:48.257501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:48.257076    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerName="maas-api" containerID="cri-o://33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9" gracePeriod=30
Nov 04 00:26:49.560499 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.560470    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:26:49.655090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.655000    6430 generic.go:356] "Generic (PLEG): container finished" podID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerID="33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9" exitCode=0
Nov 04 00:26:49.655090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.655053    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"
Nov 04 00:26:49.655090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.655079    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" event={"ID":"bae4ea80-167e-477b-a7a0-836d89dfec5a","Type":"ContainerDied","Data":"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"}
Nov 04 00:26:49.655326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.655137    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" event={"ID":"bae4ea80-167e-477b-a7a0-836d89dfec5a","Type":"ContainerDied","Data":"4f67f08704e22acac3959e01fbd1d75a4713f725449b3c0d25c69a5601ea159a"}
Nov 04 00:26:49.655326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.655167    6430 scope.go:122] "RemoveContainer" containerID="33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"
Nov 04 00:26:49.667242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.667219    6430 scope.go:122] "RemoveContainer" containerID="33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"
Nov 04 00:26:49.667632 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:26:49.667612    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9\": container with ID starting with 33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9 not found: ID does not exist" containerID="33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"
Nov 04 00:26:49.667709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.667640    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9"} err="failed to get container status \"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9\": rpc error: code = NotFound desc = could not find container \"33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9\": container with ID starting with 33f6c4452faecc3760bb4bb069bd3dafdccb9dbac5d0d7d3483cc6d8e626e4d9 not found: ID does not exist"
Nov 04 00:26:49.701487 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.701450    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") pod \"bae4ea80-167e-477b-a7a0-836d89dfec5a\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") "
Nov 04 00:26:49.701701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.701564    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\") pod \"bae4ea80-167e-477b-a7a0-836d89dfec5a\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") "
Nov 04 00:26:49.701701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.701674    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\") pod \"bae4ea80-167e-477b-a7a0-836d89dfec5a\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") "
Nov 04 00:26:49.701845 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.701728    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\" (UniqueName: \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\") pod \"bae4ea80-167e-477b-a7a0-836d89dfec5a\" (UID: \"bae4ea80-167e-477b-a7a0-836d89dfec5a\") "
Nov 04 00:26:49.701996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.701971    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca" pod "bae4ea80-167e-477b-a7a0-836d89dfec5a" (UID: "bae4ea80-167e-477b-a7a0-836d89dfec5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:26:49.702412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.702373    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae4ea80-167e-477b-a7a0-836d89dfec5a-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:49.704234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.704211    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls" pod "bae4ea80-167e-477b-a7a0-836d89dfec5a" (UID: "bae4ea80-167e-477b-a7a0-836d89dfec5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:49.704451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.704429    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs" pod "bae4ea80-167e-477b-a7a0-836d89dfec5a" (UID: "bae4ea80-167e-477b-a7a0-836d89dfec5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:26:49.704527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.704502    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd" pod "bae4ea80-167e-477b-a7a0-836d89dfec5a" (UID: "bae4ea80-167e-477b-a7a0-836d89dfec5a"). InnerVolumeSpecName "kube-api-access-wsfmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:26:49.803992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.803938    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:49.803992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.803978    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsfmd\" (UniqueName: \"kubernetes.io/projected/bae4ea80-167e-477b-a7a0-836d89dfec5a-kube-api-access-wsfmd\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:49.803992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.803991    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae4ea80-167e-477b-a7a0-836d89dfec5a-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:26:49.982246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.982207    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:26:49.986152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:49.986119    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7"]
Nov 04 00:26:50.232774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:50.232695    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" path="/var/lib/kubelet/pods/bae4ea80-167e-477b-a7a0-836d89dfec5a/volumes"
Nov 04 00:26:52.201757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:52.201707    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:26:52.490276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:26:52.490182    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d94e94-b7456f65b-468s7" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.203:8443/health\": context deadline exceeded"
Nov 04 00:27:04.863280 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:27:04.863240    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:27:09.324588 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:09.324550    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:27:09.325092 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:09.324831    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy" containerID="cri-o://d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76" gracePeriod=30
Nov 04 00:27:10.915309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.915268    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:27:10.916324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916303    6430 cpu_manager.go:414] "containerMap: removing container" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:10.916324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916323    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916335    6430 cpu_manager.go:414] "containerMap: removing container" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916340    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916361    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9a1b584f-1185-479e-b299-4017e4781746" containerName="main"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916367    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1b584f-1185-479e-b299-4017e4781746" containerName="main"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916401    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" containerName="maas-api"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916408    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" containerName="maas-api"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916425    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916431    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916446    6430 cpu_manager.go:414] "containerMap: removing container" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerName="maas-api"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916451    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerName="maas-api"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916463    6430 cpu_manager.go:414] "containerMap: removing container" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerName="main"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916468    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerName="main"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916481    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916486    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916496    6430 cpu_manager.go:414] "containerMap: removing container" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy"
Nov 04 00:27:10.916506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916500    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916603    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916614    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1b584f-1185-479e-b299-4017e4781746" containerName="main"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916625    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c50a73-86e3-4b2a-8dae-c3d482f41059" containerName="maas-api"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916633    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c5ed76-4579-4d55-88e6-d62c15b6c1fe" containerName="istio-proxy"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916642    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae4ea80-167e-477b-a7a0-836d89dfec5a" containerName="maas-api"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916648    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="178ad2bf-aad6-4ab5-a96f-53aedf955451" containerName="main"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916666    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffaa7ef3-10d9-4370-a4ac-b833cdf19e56" containerName="istio-proxy"
Nov 04 00:27:10.917085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.916672    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f5e629-bd13-4718-8a80-8c93acc775fa" containerName="istio-proxy"
Nov 04 00:27:10.922578 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.922554    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:10.925469 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.925445    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:27:10.953655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:10.953625    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c9c0227c-openshift-default-dockercfg-vskrf\""
Nov 04 00:27:11.026106 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026070    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026106 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026111    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026342 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026136    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026382 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026354    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026383    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpt2\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026525    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026565    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026582    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.026666 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.026616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.131888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.131845    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.132290 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.132263    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.132480 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.132461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.132743 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.132726    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.132858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.132844    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpt2\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.133011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.132997    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.133145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.133127    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.133270 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.133252    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.133437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.133418    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.134087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.134063    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.150901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.150862    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.152794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.151495    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.152987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.151736    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.154048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.154023    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.169458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.158675    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.189442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.185993    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.189442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.186231    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpt2\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.199760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.198731    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\") pod \"e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") " pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.253456 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.252096    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:11.573954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.573924    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:27:11.575302 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:27:11.575272    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a3a9c0_8259_49b0_848d_bac8c548a99c.slice/crio-4fb4080371d395126349efff20860036c24140bc4c9960559f46b0c53145eca2 WatchSource:0}: Error finding container 4fb4080371d395126349efff20860036c24140bc4c9960559f46b0c53145eca2: Status 404 returned error can't find the container with id 4fb4080371d395126349efff20860036c24140bc4c9960559f46b0c53145eca2
Nov 04 00:27:11.578416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.578366    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:27:11.578502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.578452    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:27:11.578502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.578481    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:27:11.991326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.991232    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerStarted","Data":"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"}
Nov 04 00:27:11.991326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:11.991298    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerStarted","Data":"4fb4080371d395126349efff20860036c24140bc4c9960559f46b0c53145eca2"}
Nov 04 00:27:12.019322 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:12.019269    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podStartSLOduration=2.019254973 podStartE2EDuration="2.019254973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:12.015734386 +0000 UTC m=+3447.906796324" watchObservedRunningTime="2026-11-04 00:27:12.019254973 +0000 UTC m=+3447.910316991"
Nov 04 00:27:12.252370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:12.252292    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:12.258511 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:12.258485    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:12.999096 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:12.999061    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:13.000095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:13.000073    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:27:14.605574 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.605509    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:27:14.694182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694145    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694230    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694299    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694375    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694456    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694554    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694604    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694671    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.694761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694670    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:14.694761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694728    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:14.694761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694746    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:14.694986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.694780    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\") pod \"96e24a1a-263c-4d7c-8737-7554bc65e822\" (UID: \"96e24a1a-263c-4d7c-8737-7554bc65e822\") "
Nov 04 00:27:14.695119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695096    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:14.695217 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:14.695421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695381    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e24a1a-263c-4d7c-8737-7554bc65e822-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.695421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695423    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.695593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695437    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.695593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695467    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.695593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.695479    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.696855 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.696825    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "kube-api-access-8t2kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:14.696964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.696916    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:14.697061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.697045    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:14.697102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.697048    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo" pod "96e24a1a-263c-4d7c-8737-7554bc65e822" (UID: "96e24a1a-263c-4d7c-8737-7554bc65e822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:27:14.797379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.797338    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.797379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.797374    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.797636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.797425    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2kr\" (UniqueName: \"kubernetes.io/projected/96e24a1a-263c-4d7c-8737-7554bc65e822-kube-api-access-8t2kr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:14.797636 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:14.797438    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e24a1a-263c-4d7c-8737-7554bc65e822-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:15.014740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.014644    6430 generic.go:356] "Generic (PLEG): container finished" podID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerID="d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76" exitCode=0
Nov 04 00:27:15.014740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.014694    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"
Nov 04 00:27:15.014971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.014726    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" event={"ID":"96e24a1a-263c-4d7c-8737-7554bc65e822","Type":"ContainerDied","Data":"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"}
Nov 04 00:27:15.014971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.014792    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn" event={"ID":"96e24a1a-263c-4d7c-8737-7554bc65e822","Type":"ContainerDied","Data":"31460ecf6b0e40de2e175ecd9565a96dc8b795ec8714bd8e7068c75281ba43b5"}
Nov 04 00:27:15.014971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.014820    6430 scope.go:122] "RemoveContainer" containerID="d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"
Nov 04 00:27:15.026514 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.026490    6430 scope.go:122] "RemoveContainer" containerID="d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"
Nov 04 00:27:15.026823 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:27:15.026802    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76\": container with ID starting with d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76 not found: ID does not exist" containerID="d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"
Nov 04 00:27:15.026887 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.026832    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76"} err="failed to get container status \"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76\": rpc error: code = NotFound desc = could not find container \"d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76\": container with ID starting with d2919ad915737a7c2880f5e53d73560cfc46a40db8169cfb00161cb13b2dbc76 not found: ID does not exist"
Nov 04 00:27:15.041336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.041302    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:27:15.044653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:15.044627    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-qr5tn"]
Nov 04 00:27:16.233957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:16.233925    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" path="/var/lib/kubelet/pods/96e24a1a-263c-4d7c-8737-7554bc65e822/volumes"
Nov 04 00:27:25.517464 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.517429    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:27:25.518367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.518343    6430 cpu_manager.go:414] "containerMap: removing container" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy"
Nov 04 00:27:25.518492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.518369    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy"
Nov 04 00:27:25.518564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.518546    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a2ad27-1b90-42ff-8977-abb226bc11c3" containerName="istio-proxy"
Nov 04 00:27:25.518626 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.518565    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e24a1a-263c-4d7c-8737-7554bc65e822" containerName="istio-proxy"
Nov 04 00:27:25.522600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.522569    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.529375 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.529348    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c9c0227c\""
Nov 04 00:27:25.529564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.529399    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c9c0227c-dockercfg-7pphz\""
Nov 04 00:27:25.530321 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.530296    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:27:25.583367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.583334    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkm9\" (UniqueName: \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.583634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.583608    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.698417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.697440    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkm9\" (UniqueName: \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.698417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.697709    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.721368 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.698811    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.724969 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.722764    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkm9\" (UniqueName: \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\") pod \"payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:25.846187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:25.846154    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:26.131163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:26.131136    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:27:26.132550 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:27:26.132523    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4991b2_f3c9_445f_b4ec_4d8a7433acb2.slice/crio-b0808c708ee224dbc2fcfbc489c165db6d53a3aac906a31fd9dc4e11784c15bb WatchSource:0}: Error finding container b0808c708ee224dbc2fcfbc489c165db6d53a3aac906a31fd9dc4e11784c15bb: Status 404 returned error can't find the container with id b0808c708ee224dbc2fcfbc489c165db6d53a3aac906a31fd9dc4e11784c15bb
Nov 04 00:27:27.143903 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:27.143849    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" event={"ID":"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2","Type":"ContainerStarted","Data":"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"}
Nov 04 00:27:27.144425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:27.143957    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:27.144425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:27.143975    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" event={"ID":"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2","Type":"ContainerStarted","Data":"b0808c708ee224dbc2fcfbc489c165db6d53a3aac906a31fd9dc4e11784c15bb"}
Nov 04 00:27:27.166059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:27.166005    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" podStartSLOduration=2.165992555 podStartE2EDuration="2.165992555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:27:27.163847636 +0000 UTC m=+3463.054909572" watchObservedRunningTime="2026-11-04 00:27:27.165992555 +0000 UTC m=+3463.057054489"
Nov 04 00:27:34.833263 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:27:34.833224    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:27:38.156793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:38.156762    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:27:39.528243 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:39.528203    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:27:45.799058 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:45.798960    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:27:45.800333 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:45.799208    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" podUID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" containerName="maas-api" containerID="cri-o://1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f" gracePeriod=30
Nov 04 00:27:46.425766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.425724    6430 generic.go:356] "Generic (PLEG): container finished" podID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" containerID="1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f" exitCode=0
Nov 04 00:27:46.430899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.430873    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:27:46.432279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.432248    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" event={"ID":"1cb0fffb-9503-4c4d-bd46-c622adf4fe98","Type":"ContainerDied","Data":"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"}
Nov 04 00:27:46.432430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.432306    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g" event={"ID":"1cb0fffb-9503-4c4d-bd46-c622adf4fe98","Type":"ContainerDied","Data":"39ccd37317353c4c924ffd3a63e5ed778dc3e8d2fab2e188ca2850629b05bd5e"}
Nov 04 00:27:46.432430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.432332    6430 scope.go:122] "RemoveContainer" containerID="1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"
Nov 04 00:27:46.443152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.443127    6430 scope.go:122] "RemoveContainer" containerID="1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"
Nov 04 00:27:46.443533 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:27:46.443502    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f\": container with ID starting with 1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f not found: ID does not exist" containerID="1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"
Nov 04 00:27:46.443653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.443545    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f"} err="failed to get container status \"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f\": rpc error: code = NotFound desc = could not find container \"1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f\": container with ID starting with 1607645826cf632708de3e59d1ee0962fe6cfaa5bb06a42f8780ef8e095ff05f not found: ID does not exist"
Nov 04 00:27:46.509417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.509299    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\") pod \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") "
Nov 04 00:27:46.509417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.509354    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\") pod \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") "
Nov 04 00:27:46.509698 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.509466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\" (UniqueName: \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\") pod \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") "
Nov 04 00:27:46.509698 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.509540    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\") pod \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\" (UID: \"1cb0fffb-9503-4c4d-bd46-c622adf4fe98\") "
Nov 04 00:27:46.509823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.509791    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca" pod "1cb0fffb-9503-4c4d-bd46-c622adf4fe98" (UID: "1cb0fffb-9503-4c4d-bd46-c622adf4fe98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:27:46.510448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.510378    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:46.511812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.511788    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs" pod "1cb0fffb-9503-4c4d-bd46-c622adf4fe98" (UID: "1cb0fffb-9503-4c4d-bd46-c622adf4fe98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:46.512279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.512250    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq" pod "1cb0fffb-9503-4c4d-bd46-c622adf4fe98" (UID: "1cb0fffb-9503-4c4d-bd46-c622adf4fe98"). InnerVolumeSpecName "kube-api-access-8kkmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:46.512448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.512382    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls" pod "1cb0fffb-9503-4c4d-bd46-c622adf4fe98" (UID: "1cb0fffb-9503-4c4d-bd46-c622adf4fe98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:46.612232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.612187    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:46.612232 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.612230    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kkmq\" (UniqueName: \"kubernetes.io/projected/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-kube-api-access-8kkmq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:46.612555 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:46.612245    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0fffb-9503-4c4d-bd46-c622adf4fe98-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:47.440710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:47.440669    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"
Nov 04 00:27:47.472380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:47.472342    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:27:47.475901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:47.475870    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecd46b-66ff9c77c-5s26g"]
Nov 04 00:27:48.234321 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:48.234288    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" path="/var/lib/kubelet/pods/1cb0fffb-9503-4c4d-bd46-c622adf4fe98/volumes"
Nov 04 00:27:54.574004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.573957    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" podUID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" containerName="main" containerID="cri-o://ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28" gracePeriod=45
Nov 04 00:27:54.882781 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.882742    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:27:54.967091 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967003    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967112    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\" (UniqueName: \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967144    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967259    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967294    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:54.967528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967372    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967500    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\") pod \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\" (UID: \"fabbe579-9d8a-44cf-a555-1ce9d2358d12\") "
Nov 04 00:27:54.967679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967652    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:54.967791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.967765    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:54.968173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.968151    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:54.968227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.968182    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:54.968227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.968199    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:54.969541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.969518    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:27:54.969613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.969584    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "kube-api-access-s2spl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:27:54.969656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:54.969633    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs" pod "fabbe579-9d8a-44cf-a555-1ce9d2358d12" (UID: "fabbe579-9d8a-44cf-a555-1ce9d2358d12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:27:55.069230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.069189    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2spl\" (UniqueName: \"kubernetes.io/projected/fabbe579-9d8a-44cf-a555-1ce9d2358d12-kube-api-access-s2spl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:55.069230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.069223    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabbe579-9d8a-44cf-a555-1ce9d2358d12-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:55.069230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.069234    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabbe579-9d8a-44cf-a555-1ce9d2358d12-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:27:55.491859 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.491826    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:27:55.518831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.518797    6430 generic.go:356] "Generic (PLEG): container finished" podID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" containerID="ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28" exitCode=0
Nov 04 00:27:55.519038 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.518848    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"
Nov 04 00:27:55.519038 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.518884    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" event={"ID":"fabbe579-9d8a-44cf-a555-1ce9d2358d12","Type":"ContainerDied","Data":"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"}
Nov 04 00:27:55.519038 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.518936    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm" event={"ID":"fabbe579-9d8a-44cf-a555-1ce9d2358d12","Type":"ContainerDied","Data":"660e4eb407671b44c480d2d1850c4a930259fd0deee4fb8bc47211da09307e20"}
Nov 04 00:27:55.519038 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.518956    6430 scope.go:122] "RemoveContainer" containerID="ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"
Nov 04 00:27:55.531982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.531958    6430 scope.go:122] "RemoveContainer" containerID="ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"
Nov 04 00:27:55.532292 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:27:55.532271    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28\": container with ID starting with ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28 not found: ID does not exist" containerID="ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"
Nov 04 00:27:55.532349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.532302    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28"} err="failed to get container status \"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28\": rpc error: code = NotFound desc = could not find container \"ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28\": container with ID starting with ee0764f984118532c792a9446557e1b20f4dd01bc9062c3af570999f04dd6c28 not found: ID does not exist"
Nov 04 00:27:55.547042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.547005    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:27:55.550279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:55.550251    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6382fc/test-model-6382fc-kserve-c84cbddc5-b89zm"]
Nov 04 00:27:56.234156 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:27:56.234118    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" path="/var/lib/kubelet/pods/fabbe579-9d8a-44cf-a555-1ce9d2358d12/volumes"
Nov 04 00:28:04.842589 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:28:04.842548    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:28:33.515420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:33.513442    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:33.528413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:33.526662    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:33.721529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:33.721471    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:33.877527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:33.877470    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:34.010527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.010477    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 04 00:28:34.113553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.113494    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:34.114227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.114190    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 04 00:28:34.140425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.140297    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 04 00:28:34.311642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.311592    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:34.397512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.396513    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 04 00:28:34.397512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.396969    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 04 00:28:34.515553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.515496    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:34.516947 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.516921    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 04 00:28:34.527436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.527353    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:34.532408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.528073    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 04 00:28:34.642121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.642087    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 04 00:28:34.821410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:34.800494    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.013098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.013053    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.042410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.014069    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 04 00:28:35.143523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.115491    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.143523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.115886    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 04 00:28:35.143523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.116490    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 04 00:28:35.143523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.116552    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 04 00:28:35.144518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.144460    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.145596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.145575    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 04 00:28:35.336413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.317787    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 04 00:28:35.336413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.318482    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 04 00:28:35.419896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.419810    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 04 00:28:35.432586 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.432555    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 04 00:28:35.432931 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.432902    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.437138 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.437112    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 04 00:28:35.498874 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.498846    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 04 00:28:35.500250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.500228    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 04 00:28:35.508714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.501079    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 04 00:28:35.546413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.538356    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:28:35.558558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.558529    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 04 00:28:35.577408 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:28:35.561200    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:28:35.584680 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:35.584646    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd"
Nov 04 00:28:42.385053 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.385017    6430 generic.go:356] "Generic (PLEG): container finished" podID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerID="d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba" exitCode=0
Nov 04 00:28:42.385548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.385087    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerDied","Data":"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"}
Nov 04 00:28:42.385598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.385579    6430 scope.go:122] "RemoveContainer" containerID="d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"
Nov 04 00:28:42.387501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.387476    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:28:42.387573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.387561    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:28:42.387611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:42.387595    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:28:43.004451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:43.000803    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:28:43.402510 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:43.402473    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerStarted","Data":"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"}
Nov 04 00:28:44.253246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:44.253214    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:28:44.254302 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:44.254276    6430 patch_prober.go:29] interesting pod/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 04 00:28:44.254460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:44.254322    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 04 00:28:45.252928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:45.252888    6430 patch_prober.go:29] interesting pod/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 04 00:28:45.253493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:45.252965    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 04 00:28:46.252660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:46.252626    6430 patch_prober.go:29] interesting pod/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 04 00:28:46.252848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:46.252710    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 04 00:28:47.252929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:47.252894    6430 patch_prober.go:29] interesting pod/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 04 00:28:47.253346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:47.252973    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 04 00:28:48.253075 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:48.253039    6430 patch_prober.go:29] interesting pod/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 04 00:28:48.253623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:48.253098    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 04 00:28:49.257594 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:49.257563    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:28:49.258030 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:49.258017    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:28:49.258615 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:28:49.258592    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:29:03.905809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:29:03.905772    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:29:04.831839 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:29:04.831793    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:29:34.831492 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:29:34.831449    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:29:54.429470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:29:54.426516    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:29:54.539419 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:29:54.507612    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:29:54.587649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:29:54.587598    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd" podUID="21599918-1626-411b-8922-0270ebf1e061" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:30:00.297861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.297823    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:00.298928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.298903    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" containerName="maas-api"
Nov 04 00:30:00.299003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.298933    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" containerName="maas-api"
Nov 04 00:30:00.299003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.298978    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" containerName="main"
Nov 04 00:30:00.299003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.298987    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" containerName="main"
Nov 04 00:30:00.299239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.299211    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb0fffb-9503-4c4d-bd46-c622adf4fe98" containerName="maas-api"
Nov 04 00:30:00.299239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.299237    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabbe579-9d8a-44cf-a555-1ce9d2358d12" containerName="main"
Nov 04 00:30:00.303934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.303904    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:00.304080 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.304046    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.309282 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.309250    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:00.309448 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.309368    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.315342 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.315305    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:00.315485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.315358    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:00.315485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.315433    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:00.315485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.315437    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.361624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361590    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.361771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361640    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghjl\" (UniqueName: \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.361815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361777    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnqf\" (UniqueName: \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.361856 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361827    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.361921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361905    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.361982 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.361969    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7mr\" (UniqueName: \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.365172 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.365150    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\""
Nov 04 00:30:00.463826 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.463790    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.464011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.463843    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghjl\" (UniqueName: \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.464011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.463917    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnqf\" (UniqueName: \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.464011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.463954    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.464011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.463983    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.464244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.464043    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7mr\" (UniqueName: \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.464675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.464650    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.464735 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.464719    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.464773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.464733    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.479184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.479145    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnqf\" (UniqueName: \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.479334 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.479187    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7mr\" (UniqueName: \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.479452 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.479345    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghjl\" (UniqueName: \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.620866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.620785    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:00.632093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.632068    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:00.638198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.638170    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:00.917671 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.917638    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:00.918271 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:30:00.918244    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8cbd67_a8c5_4333_b72c_e94997ded54a.slice/crio-a957de6a56b9d5b8165d812a34009200b413721b945c3458a2009f48742e9621 WatchSource:0}: Error finding container a957de6a56b9d5b8165d812a34009200b413721b945c3458a2009f48742e9621: Status 404 returned error can't find the container with id a957de6a56b9d5b8165d812a34009200b413721b945c3458a2009f48742e9621
Nov 04 00:30:00.974516 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.974488    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:00.975058 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:30:00.975032    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b81506_8e1c_4c9d_8067_f90d0c26f920.slice/crio-e3b6e6da9c4dc363723efc39195c0c88e7cca681ce3ebc87b7d540e1ecd37e8d WatchSource:0}: Error finding container e3b6e6da9c4dc363723efc39195c0c88e7cca681ce3ebc87b7d540e1ecd37e8d: Status 404 returned error can't find the container with id e3b6e6da9c4dc363723efc39195c0c88e7cca681ce3ebc87b7d540e1ecd37e8d
Nov 04 00:30:00.998716 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:30:00.998691    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e3984c_0695_4a78_92ef_cca9fdab2999.slice/crio-3f303bda1632befb573491b2ac4c3c5f4a8d6eea053aa1c94f398bfa1636c2fb WatchSource:0}: Error finding container 3f303bda1632befb573491b2ac4c3c5f4a8d6eea053aa1c94f398bfa1636c2fb: Status 404 returned error can't find the container with id 3f303bda1632befb573491b2ac4c3c5f4a8d6eea053aa1c94f398bfa1636c2fb
Nov 04 00:30:00.999196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:00.999177    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:01.140068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:01.140033    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerStarted","Data":"3f303bda1632befb573491b2ac4c3c5f4a8d6eea053aa1c94f398bfa1636c2fb"}
Nov 04 00:30:01.141309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:01.141285    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerStarted","Data":"a957de6a56b9d5b8165d812a34009200b413721b945c3458a2009f48742e9621"}
Nov 04 00:30:01.142501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:01.142478    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerStarted","Data":"e3b6e6da9c4dc363723efc39195c0c88e7cca681ce3ebc87b7d540e1ecd37e8d"}
Nov 04 00:30:02.151074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.151042    6430 generic.go:356] "Generic (PLEG): container finished" podID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerID="af66aea320a7bb513e5f7ac49cb56194ce75ce9db961f2bbd927b1bac322a526" exitCode=60
Nov 04 00:30:02.151629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.151129    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerDied","Data":"af66aea320a7bb513e5f7ac49cb56194ce75ce9db961f2bbd927b1bac322a526"}
Nov 04 00:30:02.151629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.151527    6430 scope.go:122] "RemoveContainer" containerID="af66aea320a7bb513e5f7ac49cb56194ce75ce9db961f2bbd927b1bac322a526"
Nov 04 00:30:02.152806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.152779    6430 generic.go:356] "Generic (PLEG): container finished" podID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerID="9d973b419437fd4789a6c3ef9390967e51da20667fb6b200b7d3f44ada243228" exitCode=60
Nov 04 00:30:02.152930 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.152807    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerDied","Data":"9d973b419437fd4789a6c3ef9390967e51da20667fb6b200b7d3f44ada243228"}
Nov 04 00:30:02.153108 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.153085    6430 scope.go:122] "RemoveContainer" containerID="9d973b419437fd4789a6c3ef9390967e51da20667fb6b200b7d3f44ada243228"
Nov 04 00:30:02.167485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.167459    6430 generic.go:356] "Generic (PLEG): container finished" podID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerID="3f4faacc40bb0b9c9e9995cd300495edf4ff303f97b317ba3462ede4da31c5cd" exitCode=60
Nov 04 00:30:02.167635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.167521    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerDied","Data":"3f4faacc40bb0b9c9e9995cd300495edf4ff303f97b317ba3462ede4da31c5cd"}
Nov 04 00:30:02.167919 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:02.167856    6430 scope.go:122] "RemoveContainer" containerID="3f4faacc40bb0b9c9e9995cd300495edf4ff303f97b317ba3462ede4da31c5cd"
Nov 04 00:30:03.176820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.176782    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerStarted","Data":"a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"}
Nov 04 00:30:03.178901 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.178868    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerStarted","Data":"594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"}
Nov 04 00:30:03.181442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.181419    6430 generic.go:356] "Generic (PLEG): container finished" podID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerID="ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1" exitCode=60
Nov 04 00:30:03.181544 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.181532    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerDied","Data":"ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1"}
Nov 04 00:30:03.181593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.181577    6430 scope.go:122] "RemoveContainer" containerID="9d973b419437fd4789a6c3ef9390967e51da20667fb6b200b7d3f44ada243228"
Nov 04 00:30:03.181822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.181800    6430 scope.go:122] "RemoveContainer" containerID="ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1"
Nov 04 00:30:03.182142 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:03.182119    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd_odh-ai-gateway-infra(9d8cbd67-a8c5-4333-b72c-e94997ded54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a"
Nov 04 00:30:03.227004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.226952    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" podStartSLOduration=3.226937185 podStartE2EDuration="3.226937185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.199024824 +0000 UTC m=+3619.090086758" watchObservedRunningTime="2026-11-04 00:30:03.226937185 +0000 UTC m=+3619.117999117"
Nov 04 00:30:03.227422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:03.227366    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" podStartSLOduration=3.22735616 podStartE2EDuration="3.22735616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.226575771 +0000 UTC m=+3619.117637711" watchObservedRunningTime="2026-11-04 00:30:03.22735616 +0000 UTC m=+3619.118418095"
Nov 04 00:30:04.190068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.190034    6430 generic.go:356] "Generic (PLEG): container finished" podID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerID="594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf" exitCode=60
Nov 04 00:30:04.190579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.190115    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerDied","Data":"594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"}
Nov 04 00:30:04.190579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.190173    6430 scope.go:122] "RemoveContainer" containerID="af66aea320a7bb513e5f7ac49cb56194ce75ce9db961f2bbd927b1bac322a526"
Nov 04 00:30:04.190579 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.190489    6430 scope.go:122] "RemoveContainer" containerID="594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"
Nov 04 00:30:04.190916 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:04.190884    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84_odh-ai-gateway-infra(f6e3984c-0695-4a78-92ef-cca9fdab2999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999"
Nov 04 00:30:04.192342 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.192325    6430 scope.go:122] "RemoveContainer" containerID="ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1"
Nov 04 00:30:04.192717 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:04.192692    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd_odh-ai-gateway-infra(9d8cbd67-a8c5-4333-b72c-e94997ded54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a"
Nov 04 00:30:04.194123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.194097    6430 generic.go:356] "Generic (PLEG): container finished" podID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerID="a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369" exitCode=60
Nov 04 00:30:04.194244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.194144    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerDied","Data":"a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"}
Nov 04 00:30:04.194649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.194626    6430 scope.go:122] "RemoveContainer" containerID="a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"
Nov 04 00:30:04.194983 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:04.194962    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj_odh-ai-gateway-infra(67b81506-8e1c-4c9d-8067-f90d0c26f920)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920"
Nov 04 00:30:04.205851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.205825    6430 scope.go:122] "RemoveContainer" containerID="3f4faacc40bb0b9c9e9995cd300495edf4ff303f97b317ba3462ede4da31c5cd"
Nov 04 00:30:04.526969 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:04.526786    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:30:04.834885 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:04.834488    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:30:05.203733 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:05.203641    6430 scope.go:122] "RemoveContainer" containerID="594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"
Nov 04 00:30:05.204125 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:05.204041    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84_odh-ai-gateway-infra(f6e3984c-0695-4a78-92ef-cca9fdab2999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999"
Nov 04 00:30:05.206493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:05.206468    6430 scope.go:122] "RemoveContainer" containerID="a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"
Nov 04 00:30:05.206876 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:05.206852    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj_odh-ai-gateway-infra(67b81506-8e1c-4c9d-8067-f90d0c26f920)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920"
Nov 04 00:30:09.530506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.528049    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" podUID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" containerName="payload-pre-processing" containerID="cri-o://6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e" gracePeriod=25
Nov 04 00:30:09.822250 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.822220    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:30:09.928352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.928311    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\" (UniqueName: \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\") pod \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") "
Nov 04 00:30:09.928551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.928466    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\") pod \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\" (UID: \"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2\") "
Nov 04 00:30:09.928831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.928805    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume" pod "fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" (UID: "fd4991b2-f3c9-445f-b4ec-4d8a7433acb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:09.929061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.929043    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:09.930516 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:09.930489    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9" pod "fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" (UID: "fd4991b2-f3c9-445f-b4ec-4d8a7433acb2"). InnerVolumeSpecName "kube-api-access-nqkm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:10.030172 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.030125    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkm9\" (UniqueName: \"kubernetes.io/projected/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2-kube-api-access-nqkm9\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:10.241995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.241956    6430 generic.go:356] "Generic (PLEG): container finished" podID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" containerID="6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e" exitCode=0
Nov 04 00:30:10.242176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.242010    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" event={"ID":"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2","Type":"ContainerDied","Data":"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"}
Nov 04 00:30:10.242176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.242024    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"
Nov 04 00:30:10.242176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.242055    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2" event={"ID":"fd4991b2-f3c9-445f-b4ec-4d8a7433acb2","Type":"ContainerDied","Data":"b0808c708ee224dbc2fcfbc489c165db6d53a3aac906a31fd9dc4e11784c15bb"}
Nov 04 00:30:10.242176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.242079    6430 scope.go:122] "RemoveContainer" containerID="6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"
Nov 04 00:30:10.254099 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.254081    6430 scope.go:122] "RemoveContainer" containerID="6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"
Nov 04 00:30:10.254407 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:10.254366    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e\": container with ID starting with 6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e not found: ID does not exist" containerID="6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"
Nov 04 00:30:10.254493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.254407    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e"} err="failed to get container status \"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e\": rpc error: code = NotFound desc = could not find container \"6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e\": container with ID starting with 6806cac0f14bc94147ce676622333b4eaa29aeb555e027c14d66ffa615ede73e not found: ID does not exist"
Nov 04 00:30:10.299437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.293921    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:30:10.302601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:10.302542    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9c0227c-8494dd8c75-fv4g2"]
Nov 04 00:30:12.235519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:12.234249    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" path="/var/lib/kubelet/pods/fd4991b2-f3c9-445f-b4ec-4d8a7433acb2/volumes"
Nov 04 00:30:14.229158 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:14.229124    6430 scope.go:122] "RemoveContainer" containerID="594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"
Nov 04 00:30:15.281429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:15.281376    6430 generic.go:356] "Generic (PLEG): container finished" podID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerID="b66a832b508400c2e2d2d17d7c21ebc45baa22b729ada255c9aa5b92780af078" exitCode=60
Nov 04 00:30:15.281909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:15.281436    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerDied","Data":"b66a832b508400c2e2d2d17d7c21ebc45baa22b729ada255c9aa5b92780af078"}
Nov 04 00:30:15.281909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:15.281521    6430 scope.go:122] "RemoveContainer" containerID="594832d02b912d439f64d85dbb2c5a29fa945b8238e5fea6f6f49cd6c182ddaf"
Nov 04 00:30:15.281909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:15.281894    6430 scope.go:122] "RemoveContainer" containerID="b66a832b508400c2e2d2d17d7c21ebc45baa22b729ada255c9aa5b92780af078"
Nov 04 00:30:15.282269 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:15.282246    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84_odh-ai-gateway-infra(f6e3984c-0695-4a78-92ef-cca9fdab2999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999"
Nov 04 00:30:16.224956 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.224919    6430 scope.go:122] "RemoveContainer" containerID="a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"
Nov 04 00:30:16.313352 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.313321    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:16.571566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.571533    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:16.656062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.656026    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\") pod \"f6e3984c-0695-4a78-92ef-cca9fdab2999\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") "
Nov 04 00:30:16.656062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.656065    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\" (UniqueName: \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\") pod \"f6e3984c-0695-4a78-92ef-cca9fdab2999\" (UID: \"f6e3984c-0695-4a78-92ef-cca9fdab2999\") "
Nov 04 00:30:16.656471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.656442    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca" pod "f6e3984c-0695-4a78-92ef-cca9fdab2999" (UID: "f6e3984c-0695-4a78-92ef-cca9fdab2999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:16.656808 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.656788    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e3984c-0695-4a78-92ef-cca9fdab2999-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:16.658341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.658311    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf" pod "f6e3984c-0695-4a78-92ef-cca9fdab2999" (UID: "f6e3984c-0695-4a78-92ef-cca9fdab2999"). InnerVolumeSpecName "kube-api-access-llnqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:16.758061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:16.758029    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnqf\" (UniqueName: \"kubernetes.io/projected/f6e3984c-0695-4a78-92ef-cca9fdab2999-kube-api-access-llnqf\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:17.227230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.226602    6430 scope.go:122] "RemoveContainer" containerID="ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1"
Nov 04 00:30:17.312672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.311249    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerStarted","Data":"9cdb76438fe6097905116cd772ba54fb3ce2d678f7ec9da55c5d1f8cc982bf71"}
Nov 04 00:30:17.318687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.318649    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84" event={"ID":"f6e3984c-0695-4a78-92ef-cca9fdab2999","Type":"ContainerDied","Data":"3f303bda1632befb573491b2ac4c3c5f4a8d6eea053aa1c94f398bfa1636c2fb"}
Nov 04 00:30:17.319140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.318771    6430 scope.go:122] "RemoveContainer" containerID="b66a832b508400c2e2d2d17d7c21ebc45baa22b729ada255c9aa5b92780af078"
Nov 04 00:30:17.319140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.318954    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"
Nov 04 00:30:17.377617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.377580    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:17.381493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:17.381463    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-398158-29895870-d7r84"]
Nov 04 00:30:18.234504 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.234469    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" path="/var/lib/kubelet/pods/f6e3984c-0695-4a78-92ef-cca9fdab2999/volumes"
Nov 04 00:30:18.329543 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.329512    6430 generic.go:356] "Generic (PLEG): container finished" podID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerID="9cdb76438fe6097905116cd772ba54fb3ce2d678f7ec9da55c5d1f8cc982bf71" exitCode=60
Nov 04 00:30:18.329923 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.329580    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerDied","Data":"9cdb76438fe6097905116cd772ba54fb3ce2d678f7ec9da55c5d1f8cc982bf71"}
Nov 04 00:30:18.329923 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.329647    6430 scope.go:122] "RemoveContainer" containerID="a64260e5404bf2a83ac8265a1311740fccd0c661533f773b6e1fb1f4ac2e2369"
Nov 04 00:30:18.330153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.330128    6430 scope.go:122] "RemoveContainer" containerID="9cdb76438fe6097905116cd772ba54fb3ce2d678f7ec9da55c5d1f8cc982bf71"
Nov 04 00:30:18.330859 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:18.330620    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj_odh-ai-gateway-infra(67b81506-8e1c-4c9d-8067-f90d0c26f920)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920"
Nov 04 00:30:18.333020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.332994    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerStarted","Data":"0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b"}
Nov 04 00:30:18.333422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.333406    6430 scope.go:122] "RemoveContainer" containerID="0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b"
Nov 04 00:30:18.333761 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:18.333740    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd_odh-ai-gateway-infra(9d8cbd67-a8c5-4333-b72c-e94997ded54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a"
Nov 04 00:30:18.350240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:18.350210    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:19.344261 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.344226    6430 generic.go:356] "Generic (PLEG): container finished" podID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerID="0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b" exitCode=60
Nov 04 00:30:19.344590 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.344296    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerDied","Data":"0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b"}
Nov 04 00:30:19.344590 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.344365    6430 scope.go:122] "RemoveContainer" containerID="ba5bd97357da863d17753aba4092c70a5888d96b71cad131c59bc162c10971c1"
Nov 04 00:30:19.344825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.344805    6430 scope.go:122] "RemoveContainer" containerID="0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b"
Nov 04 00:30:19.345201 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:19.345171    6430 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd_odh-ai-gateway-infra(9d8cbd67-a8c5-4333-b72c-e94997ded54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a"
Nov 04 00:30:19.397028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.396995    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:19.507186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.507133    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:19.600657 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.600556    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\" (UniqueName: \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\") pod \"67b81506-8e1c-4c9d-8067-f90d0c26f920\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") "
Nov 04 00:30:19.600835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.600675    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\") pod \"67b81506-8e1c-4c9d-8067-f90d0c26f920\" (UID: \"67b81506-8e1c-4c9d-8067-f90d0c26f920\") "
Nov 04 00:30:19.601089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.601052    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca" pod "67b81506-8e1c-4c9d-8067-f90d0c26f920" (UID: "67b81506-8e1c-4c9d-8067-f90d0c26f920"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:19.601478 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.601457    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b81506-8e1c-4c9d-8067-f90d0c26f920-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:19.603073 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.603048    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl" pod "67b81506-8e1c-4c9d-8067-f90d0c26f920" (UID: "67b81506-8e1c-4c9d-8067-f90d0c26f920"). InnerVolumeSpecName "kube-api-access-nghjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:19.702836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:19.702799    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghjl\" (UniqueName: \"kubernetes.io/projected/67b81506-8e1c-4c9d-8067-f90d0c26f920-kube-api-access-nghjl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:20.355632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.355601    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj" event={"ID":"67b81506-8e1c-4c9d-8067-f90d0c26f920","Type":"ContainerDied","Data":"e3b6e6da9c4dc363723efc39195c0c88e7cca681ce3ebc87b7d540e1ecd37e8d"}
Nov 04 00:30:20.355632 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.355622    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"
Nov 04 00:30:20.356083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.355664    6430 scope.go:122] "RemoveContainer" containerID="9cdb76438fe6097905116cd772ba54fb3ce2d678f7ec9da55c5d1f8cc982bf71"
Nov 04 00:30:20.400611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.400575    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:20.407202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.407171    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895870-h24tj"]
Nov 04 00:30:20.529285 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.529256    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:20.622851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.622815    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\" (UniqueName: \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\") pod \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") "
Nov 04 00:30:20.623021 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.622903    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\") pod \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\" (UID: \"9d8cbd67-a8c5-4333-b72c-e94997ded54a\") "
Nov 04 00:30:20.623257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.623231    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca" pod "9d8cbd67-a8c5-4333-b72c-e94997ded54a" (UID: "9d8cbd67-a8c5-4333-b72c-e94997ded54a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:20.623646 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.623627    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8cbd67-a8c5-4333-b72c-e94997ded54a-maas-api-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:20.624911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.624888    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr" pod "9d8cbd67-a8c5-4333-b72c-e94997ded54a" (UID: "9d8cbd67-a8c5-4333-b72c-e94997ded54a"). InnerVolumeSpecName "kube-api-access-mx7mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:20.725519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:20.725486    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7mr\" (UniqueName: \"kubernetes.io/projected/9d8cbd67-a8c5-4333-b72c-e94997ded54a-kube-api-access-mx7mr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:21.363847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.363814    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"
Nov 04 00:30:21.364441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.363841    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd" event={"ID":"9d8cbd67-a8c5-4333-b72c-e94997ded54a","Type":"ContainerDied","Data":"a957de6a56b9d5b8165d812a34009200b413721b945c3458a2009f48742e9621"}
Nov 04 00:30:21.364441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.363910    6430 scope.go:122] "RemoveContainer" containerID="0bd98afc8120460b1d4394e987f63a35cf9f19d6c8bdbee0d0b5a4d320fa7c1b"
Nov 04 00:30:21.395253 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.395219    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:21.400723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.400696    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895870-d6tkd"]
Nov 04 00:30:21.821852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:21.821815    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:30:22.233967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:22.233886    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" path="/var/lib/kubelet/pods/67b81506-8e1c-4c9d-8067-f90d0c26f920/volumes"
Nov 04 00:30:22.234441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:22.234425    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" path="/var/lib/kubelet/pods/9d8cbd67-a8c5-4333-b72c-e94997ded54a/volumes"
Nov 04 00:30:34.832969 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:34.832933    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:30:36.423954 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:36.423907    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:30:36.424825 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:36.424787    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy" containerID="cri-o://67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4" gracePeriod=30
Nov 04 00:30:41.694532 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.694458    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:30:41.797273 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797229    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797432    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797467    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797485    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797527    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797559    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797593    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797682    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\") pod \"94a3a9c0-8259-49b0-848d-bac8c548a99c\" (UID: \"94a3a9c0-8259-49b0-848d-bac8c548a99c\") "
Nov 04 00:30:41.797951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797823    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:30:41.797951 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.797842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:30:41.798069 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798029    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:30:41.798069 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798037    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:30:41.798350 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798326    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:30:41.798625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798600    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.798713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798631    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.798713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798647    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.798713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798663    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94a3a9c0-8259-49b0-848d-bac8c548a99c-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.798713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.798678    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.799869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.799842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:30:41.800036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.800017    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:41.800114 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.800095    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "kube-api-access-rzpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:30:41.800164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.800092    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo" pod "94a3a9c0-8259-49b0-848d-bac8c548a99c" (UID: "94a3a9c0-8259-49b0-848d-bac8c548a99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:30:41.899812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.899781    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.899812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.899809    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpt2\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-kube-api-access-rzpt2\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.899812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.899821    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:41.900055 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:41.899829    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94a3a9c0-8259-49b0-848d-bac8c548a99c-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:30:42.538373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.538285    6430 generic.go:356] "Generic (PLEG): container finished" podID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerID="67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4" exitCode=0
Nov 04 00:30:42.538373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.538350    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"
Nov 04 00:30:42.538373 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.538362    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerDied","Data":"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"}
Nov 04 00:30:42.538619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.538469    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc" event={"ID":"94a3a9c0-8259-49b0-848d-bac8c548a99c","Type":"ContainerDied","Data":"4fb4080371d395126349efff20860036c24140bc4c9960559f46b0c53145eca2"}
Nov 04 00:30:42.538619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.538494    6430 scope.go:122] "RemoveContainer" containerID="67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"
Nov 04 00:30:42.549367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.549344    6430 scope.go:122] "RemoveContainer" containerID="d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"
Nov 04 00:30:42.559817 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.559622    6430 scope.go:122] "RemoveContainer" containerID="67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"
Nov 04 00:30:42.560003 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:42.559984    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4\": container with ID starting with 67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4 not found: ID does not exist" containerID="67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"
Nov 04 00:30:42.560071 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.560012    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4"} err="failed to get container status \"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4\": rpc error: code = NotFound desc = could not find container \"67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4\": container with ID starting with 67441fb754705cb49a994daa41bbc1607c9a5da11f09f46abf13450b0ed9a1f4 not found: ID does not exist"
Nov 04 00:30:42.560071 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.560029    6430 scope.go:122] "RemoveContainer" containerID="d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"
Nov 04 00:30:42.560310 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:30:42.560292    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba\": container with ID starting with d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba not found: ID does not exist" containerID="d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"
Nov 04 00:30:42.560358 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.560316    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba"} err="failed to get container status \"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba\": rpc error: code = NotFound desc = could not find container \"d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba\": container with ID starting with d5d8a0fed3dbb142d8229c4e29373baa6941b925b652ff0ccacc0da5267e31ba not found: ID does not exist"
Nov 04 00:30:42.569539 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.566551    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:30:42.570623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:42.570599    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c9c0227c-openshift-default-6bbd88fbb5-lrhvc"]
Nov 04 00:30:44.233599 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:30:44.233566    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" path="/var/lib/kubelet/pods/94a3a9c0-8259-49b0-848d-bac8c548a99c/volumes"
Nov 04 00:31:01.179656 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:01.179611    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:31:01.180198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:01.179902    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy" containerID="cri-o://6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914" gracePeriod=30
Nov 04 00:31:01.422357 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:01.422317    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:31:04.831917 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:04.831869    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:31:06.471442 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.471359    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:31:06.521964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.521922    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522148 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522042    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522148 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522101    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522131    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522244    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:06.522473 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522449    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:06.522540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522678    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522713    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522771    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.522836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522827    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\") pod \"8c76493c-8efd-42a4-90d6-cefa92b7788e\" (UID: \"8c76493c-8efd-42a4-90d6-cefa92b7788e\") "
Nov 04 00:31:06.523014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.522989    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:06.523438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.523377    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:06.523725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.523586    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.523809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.523737    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.523809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.523756    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.523809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.523681    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:06.524966 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.524938    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:06.525548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.525517    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "kube-api-access-5pf4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:06.526568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.526545    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:31:06.526854 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.526826    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token" pod "8c76493c-8efd-42a4-90d6-cefa92b7788e" (UID: "8c76493c-8efd-42a4-90d6-cefa92b7788e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:06.624597 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624548    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.624597 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624588    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.624597 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624605    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pf4j\" (UniqueName: \"kubernetes.io/projected/8c76493c-8efd-42a4-90d6-cefa92b7788e-kube-api-access-5pf4j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.624844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624618    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c76493c-8efd-42a4-90d6-cefa92b7788e-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.624844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624629    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c76493c-8efd-42a4-90d6-cefa92b7788e-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.624844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.624639    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c76493c-8efd-42a4-90d6-cefa92b7788e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:06.718176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.718130    6430 generic.go:356] "Generic (PLEG): container finished" podID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerID="6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914" exitCode=0
Nov 04 00:31:06.718374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.718179    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"
Nov 04 00:31:06.718374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.718201    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerDied","Data":"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"}
Nov 04 00:31:06.718374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.718256    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f" event={"ID":"8c76493c-8efd-42a4-90d6-cefa92b7788e","Type":"ContainerDied","Data":"d34d206a0aaa868156b80ca09e572b4494435e1253876f94fc3337852d4878bf"}
Nov 04 00:31:06.718374 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.718278    6430 scope.go:122] "RemoveContainer" containerID="6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"
Nov 04 00:31:06.729975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.729957    6430 scope.go:122] "RemoveContainer" containerID="5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"
Nov 04 00:31:06.739227 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.739208    6430 scope.go:122] "RemoveContainer" containerID="6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"
Nov 04 00:31:06.739531 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:06.739510    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914\": container with ID starting with 6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914 not found: ID does not exist" containerID="6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"
Nov 04 00:31:06.739600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.739539    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914"} err="failed to get container status \"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914\": rpc error: code = NotFound desc = could not find container \"6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914\": container with ID starting with 6d7215c57d6c4e5916304f58261443f9117d6fbc166c334a2a0442ca19950914 not found: ID does not exist"
Nov 04 00:31:06.739600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.739555    6430 scope.go:122] "RemoveContainer" containerID="5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"
Nov 04 00:31:06.739803 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:06.739782    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd\": container with ID starting with 5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd not found: ID does not exist" containerID="5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"
Nov 04 00:31:06.739868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.739813    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd"} err="failed to get container status \"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd\": rpc error: code = NotFound desc = could not find container \"5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd\": container with ID starting with 5745c9cd0fdfadd96303ec91effbbc22700a8e8faf77690543f8b53532fd1afd not found: ID does not exist"
Nov 04 00:31:06.745428 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.743877    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:31:06.747700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:06.747681    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a002b-openshift-default-7fdc9845df-srj8f"]
Nov 04 00:31:08.234093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:08.234057    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" path="/var/lib/kubelet/pods/8c76493c-8efd-42a4-90d6-cefa92b7788e/volumes"
Nov 04 00:31:26.418872 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:26.418822    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 04 00:31:26.425124 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:26.425088    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 04 00:31:31.419855 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.419783    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" podUID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" containerName="payload-processing" containerID="cri-o://ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603" gracePeriod=25
Nov 04 00:31:31.425361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.425332    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" podUID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" containerName="payload-pre-processing" containerID="cri-o://c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219" gracePeriod=25
Nov 04 00:31:31.714791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.714761    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 04 00:31:31.750435 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.750379    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 04 00:31:31.838305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838268    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\") pod \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") "
Nov 04 00:31:31.838563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838336    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\") pod \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") "
Nov 04 00:31:31.838563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838433    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\") pod \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\" (UID: \"9561d9a3-02b4-43a6-a1a7-4cab5c54a311\") "
Nov 04 00:31:31.838739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838718    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\" (UniqueName: \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\") pod \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\" (UID: \"dfb20907-450f-42e1-bfc6-c560ea4ab7fd\") "
Nov 04 00:31:31.838842 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838820    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume" pod "9561d9a3-02b4-43a6-a1a7-4cab5c54a311" (UID: "9561d9a3-02b4-43a6-a1a7-4cab5c54a311"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:31.838889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.838824    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume" pod "dfb20907-450f-42e1-bfc6-c560ea4ab7fd" (UID: "dfb20907-450f-42e1-bfc6-c560ea4ab7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:31.839201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.839175    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:31.839201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.839196    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:31.840640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.840617    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb" pod "9561d9a3-02b4-43a6-a1a7-4cab5c54a311" (UID: "9561d9a3-02b4-43a6-a1a7-4cab5c54a311"). InnerVolumeSpecName "kube-api-access-h6zdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:31.840769 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.840746    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7" pod "dfb20907-450f-42e1-bfc6-c560ea4ab7fd" (UID: "dfb20907-450f-42e1-bfc6-c560ea4ab7fd"). InnerVolumeSpecName "kube-api-access-jhqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:31.941261 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.941172    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqj7\" (UniqueName: \"kubernetes.io/projected/dfb20907-450f-42e1-bfc6-c560ea4ab7fd-kube-api-access-jhqj7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:31.941261 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.941208    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/9561d9a3-02b4-43a6-a1a7-4cab5c54a311-kube-api-access-h6zdb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:31.977264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.977193    6430 generic.go:356] "Generic (PLEG): container finished" podID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" containerID="c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219" exitCode=0
Nov 04 00:31:31.977264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.977254    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"
Nov 04 00:31:31.977605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.977423    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" event={"ID":"dfb20907-450f-42e1-bfc6-c560ea4ab7fd","Type":"ContainerDied","Data":"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"}
Nov 04 00:31:31.977605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.977507    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt" event={"ID":"dfb20907-450f-42e1-bfc6-c560ea4ab7fd","Type":"ContainerDied","Data":"52926a6aa2124888f2a4f8d19cc82b78f9f95cde1a782b7e88e2d7603901eaba"}
Nov 04 00:31:31.977605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.977537    6430 scope.go:122] "RemoveContainer" containerID="c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"
Nov 04 00:31:31.979036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.979008    6430 generic.go:356] "Generic (PLEG): container finished" podID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" containerID="ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603" exitCode=0
Nov 04 00:31:31.979166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.979056    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" event={"ID":"9561d9a3-02b4-43a6-a1a7-4cab5c54a311","Type":"ContainerDied","Data":"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"}
Nov 04 00:31:31.979166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.979063    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"
Nov 04 00:31:31.979166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.979094    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s" event={"ID":"9561d9a3-02b4-43a6-a1a7-4cab5c54a311","Type":"ContainerDied","Data":"88c4ab9b8be3c23687fca652e0f262f370c22c4c033e415d616fd9798094fbb8"}
Nov 04 00:31:31.987701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.987676    6430 scope.go:122] "RemoveContainer" containerID="c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"
Nov 04 00:31:31.988018 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:31.987993    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219\": container with ID starting with c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219 not found: ID does not exist" containerID="c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"
Nov 04 00:31:31.988125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.988030    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219"} err="failed to get container status \"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219\": rpc error: code = NotFound desc = could not find container \"c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219\": container with ID starting with c80e0ba4d0fe78f42ff5a69a28a4a2b32ec9141b86de3cc6d065ca0ac442a219 not found: ID does not exist"
Nov 04 00:31:31.988125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.988051    6430 scope.go:122] "RemoveContainer" containerID="ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"
Nov 04 00:31:31.998756 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.998738    6430 scope.go:122] "RemoveContainer" containerID="ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"
Nov 04 00:31:31.999032 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:31.999011    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603\": container with ID starting with ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603 not found: ID does not exist" containerID="ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"
Nov 04 00:31:31.999085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:31.999039    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603"} err="failed to get container status \"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603\": rpc error: code = NotFound desc = could not find container \"ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603\": container with ID starting with ec28f0a1247746163cefeb9a7f4a8de424245ac9dec9005f3bb4e7d17f8d6603 not found: ID does not exist"
Nov 04 00:31:32.010290 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.010260    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 04 00:31:32.015844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.015816    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-330eb5-795846b8d7-pczgt"]
Nov 04 00:31:32.028891 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.028849    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 04 00:31:32.034295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.034263    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-330eb5-6c7bcb655b-lcs5s"]
Nov 04 00:31:32.257896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.257809    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" path="/var/lib/kubelet/pods/9561d9a3-02b4-43a6-a1a7-4cab5c54a311/volumes"
Nov 04 00:31:32.258333 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:32.258317    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" path="/var/lib/kubelet/pods/dfb20907-450f-42e1-bfc6-c560ea4ab7fd/volumes"
Nov 04 00:31:34.832717 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:34.832675    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:31:42.217427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.217367    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 04 00:31:42.218125 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.217672    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" podUID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" containerName="maas-api" containerID="cri-o://17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001" gracePeriod=30
Nov 04 00:31:42.490767 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.490735    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 04 00:31:42.649178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.649132    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\") pod \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") "
Nov 04 00:31:42.649411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.649218    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\" (UniqueName: \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\") pod \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") "
Nov 04 00:31:42.649488 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.649448    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\") pod \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") "
Nov 04 00:31:42.649568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.649547    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\") pod \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\" (UID: \"4bc82e6b-c5d9-4863-8712-70ebc2236f8c\") "
Nov 04 00:31:42.676541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.676498    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca" pod "4bc82e6b-c5d9-4863-8712-70ebc2236f8c" (UID: "4bc82e6b-c5d9-4863-8712-70ebc2236f8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:42.715645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.715603    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs" pod "4bc82e6b-c5d9-4863-8712-70ebc2236f8c" (UID: "4bc82e6b-c5d9-4863-8712-70ebc2236f8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:31:42.716230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.716201    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls" pod "4bc82e6b-c5d9-4863-8712-70ebc2236f8c" (UID: "4bc82e6b-c5d9-4863-8712-70ebc2236f8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:31:42.716767 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.716738    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7" pod "4bc82e6b-c5d9-4863-8712-70ebc2236f8c" (UID: "4bc82e6b-c5d9-4863-8712-70ebc2236f8c"). InnerVolumeSpecName "kube-api-access-xvnw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:42.752819 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.752727    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:42.753059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.753035    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnw7\" (UniqueName: \"kubernetes.io/projected/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-kube-api-access-xvnw7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:42.753181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.753166    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:42.753289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:42.753277    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc82e6b-c5d9-4863-8712-70ebc2236f8c-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:43.057690 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.057648    6430 generic.go:356] "Generic (PLEG): container finished" podID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" containerID="17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001" exitCode=0
Nov 04 00:31:43.057880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.057700    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"
Nov 04 00:31:43.057880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.057735    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" event={"ID":"4bc82e6b-c5d9-4863-8712-70ebc2236f8c","Type":"ContainerDied","Data":"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"}
Nov 04 00:31:43.057880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.057791    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv" event={"ID":"4bc82e6b-c5d9-4863-8712-70ebc2236f8c","Type":"ContainerDied","Data":"b2a53170276e939082f36d12f6cf90bf84106ebc51c51c1b978ee4ce93881267"}
Nov 04 00:31:43.057880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.057816    6430 scope.go:122] "RemoveContainer" containerID="17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"
Nov 04 00:31:43.068239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.068214    6430 scope.go:122] "RemoveContainer" containerID="17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"
Nov 04 00:31:43.068530 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:43.068511    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001\": container with ID starting with 17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001 not found: ID does not exist" containerID="17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"
Nov 04 00:31:43.068602 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.068537    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001"} err="failed to get container status \"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001\": rpc error: code = NotFound desc = could not find container \"17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001\": container with ID starting with 17b0b2267826f32f5d20242e3193a8ee6982fe609b26e9600fc7c0da19886001 not found: ID does not exist"
Nov 04 00:31:43.082884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.082855    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 04 00:31:43.088364 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:43.088337    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-330eb5-64bc79cdf7-kvxxv"]
Nov 04 00:31:44.234274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:44.234242    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" path="/var/lib/kubelet/pods/4bc82e6b-c5d9-4863-8712-70ebc2236f8c/volumes"
Nov 04 00:31:47.718248 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:47.718208    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:31:53.179151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.176177    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:31:53.179151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.177161    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api" containerID="cri-o://49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b" gracePeriod=30
Nov 04 00:31:53.275868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.275821    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 04 00:31:53.276370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.276332    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy" containerID="cri-o://a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532" gracePeriod=30
Nov 04 00:31:53.556700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.556489    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:31:53.622790 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.622746    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\") pod \"61a79208-d133-4c26-90dd-364f5146ee8a\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") "
Nov 04 00:31:53.623002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.622854    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\") pod \"61a79208-d133-4c26-90dd-364f5146ee8a\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") "
Nov 04 00:31:53.623002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.622971    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\") pod \"61a79208-d133-4c26-90dd-364f5146ee8a\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") "
Nov 04 00:31:53.623139 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.623026    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\" (UniqueName: \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\") pod \"61a79208-d133-4c26-90dd-364f5146ee8a\" (UID: \"61a79208-d133-4c26-90dd-364f5146ee8a\") "
Nov 04 00:31:53.623198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.623168    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca" pod "61a79208-d133-4c26-90dd-364f5146ee8a" (UID: "61a79208-d133-4c26-90dd-364f5146ee8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:53.623792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.623749    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a79208-d133-4c26-90dd-364f5146ee8a-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:53.625613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.625577    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs" pod "61a79208-d133-4c26-90dd-364f5146ee8a" (UID: "61a79208-d133-4c26-90dd-364f5146ee8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:31:53.625705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.625623    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls" pod "61a79208-d133-4c26-90dd-364f5146ee8a" (UID: "61a79208-d133-4c26-90dd-364f5146ee8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:31:53.625705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.625630    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq" pod "61a79208-d133-4c26-90dd-364f5146ee8a" (UID: "61a79208-d133-4c26-90dd-364f5146ee8a"). InnerVolumeSpecName "kube-api-access-qn2qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:53.724793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.724751    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:53.724793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.724787    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61a79208-d133-4c26-90dd-364f5146ee8a-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:53.724793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:53.724797    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn2qq\" (UniqueName: \"kubernetes.io/projected/61a79208-d133-4c26-90dd-364f5146ee8a-kube-api-access-qn2qq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:54.143625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.143586    6430 generic.go:356] "Generic (PLEG): container finished" podID="61a79208-d133-4c26-90dd-364f5146ee8a" containerID="49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b" exitCode=0
Nov 04 00:31:54.143820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.143638    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"
Nov 04 00:31:54.143820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.143667    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerDied","Data":"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"}
Nov 04 00:31:54.143820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.143722    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895" event={"ID":"61a79208-d133-4c26-90dd-364f5146ee8a","Type":"ContainerDied","Data":"2bba8d5b767c03ceb949d0cdcc5d357bdd8d5e7163a881c0c37be0ce4ec1be91"}
Nov 04 00:31:54.143820 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.143744    6430 scope.go:122] "RemoveContainer" containerID="49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"
Nov 04 00:31:54.155646 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.155624    6430 scope.go:122] "RemoveContainer" containerID="9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"
Nov 04 00:31:54.166019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.165999    6430 scope.go:122] "RemoveContainer" containerID="49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"
Nov 04 00:31:54.166359 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:54.166333    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b\": container with ID starting with 49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b not found: ID does not exist" containerID="49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"
Nov 04 00:31:54.166479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.166368    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b"} err="failed to get container status \"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b\": rpc error: code = NotFound desc = could not find container \"49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b\": container with ID starting with 49a6883c70bc852b2e841aa0a89d51e8f8b058390debec2580da258bfbc3765b not found: ID does not exist"
Nov 04 00:31:54.166479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.166407    6430 scope.go:122] "RemoveContainer" containerID="9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"
Nov 04 00:31:54.166807 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:54.166778    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde\": container with ID starting with 9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde not found: ID does not exist" containerID="9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"
Nov 04 00:31:54.166993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.166815    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde"} err="failed to get container status \"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde\": rpc error: code = NotFound desc = could not find container \"9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde\": container with ID starting with 9a86058435a58e32adb11415e20bced722b0ea13c93b2fd79cfe84d6dc848fde not found: ID does not exist"
Nov 04 00:31:54.170854 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.170829    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:31:54.176591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.176566    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-71b2f3-6cb94589d4-22895"]
Nov 04 00:31:54.236130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:54.233515    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" path="/var/lib/kubelet/pods/61a79208-d133-4c26-90dd-364f5146ee8a/volumes"
Nov 04 00:31:58.586294 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.586265    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 04 00:31:58.697192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.697143    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.697573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.697537    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.697897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.697859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.697940    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698350 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698327    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698407    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698599    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698651    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.698786 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698712    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\") pod \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\" (UID: \"25d3f943-66dd-4eea-a7c0-b6969c824c1b\") "
Nov 04 00:31:58.699066 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698832    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:58.702957 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.698903    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:31:58.703482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.703429    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:58.703947 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.703648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:58.704103 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.703691    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704681    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704816    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704845    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704869    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704883    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.705377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.704898    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.721449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.711867    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:31:58.730898 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.730855    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:58.731083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.730969    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx" pod "25d3f943-66dd-4eea-a7c0-b6969c824c1b" (UID: "25d3f943-66dd-4eea-a7c0-b6969c824c1b"). InnerVolumeSpecName "kube-api-access-rcqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:31:58.806470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.806364    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqmx\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-kube-api-access-rcqmx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.806470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.806423    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.806470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.806434    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:58.806470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:58.806443    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d3f943-66dd-4eea-a7c0-b6969c824c1b-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:31:59.184706 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.184617    6430 generic.go:356] "Generic (PLEG): container finished" podID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerID="a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532" exitCode=0
Nov 04 00:31:59.184706 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.184689    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"
Nov 04 00:31:59.185031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.184698    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerDied","Data":"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"}
Nov 04 00:31:59.185031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.184790    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n" event={"ID":"25d3f943-66dd-4eea-a7c0-b6969c824c1b","Type":"ContainerDied","Data":"1f6ace4a056dd330d4207356a6576d0d07e495fd98880017aaa4a538580ef77a"}
Nov 04 00:31:59.185031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.184822    6430 scope.go:122] "RemoveContainer" containerID="a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"
Nov 04 00:31:59.195851 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.195825    6430 scope.go:122] "RemoveContainer" containerID="d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"
Nov 04 00:31:59.206035 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.206012    6430 scope.go:122] "RemoveContainer" containerID="a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"
Nov 04 00:31:59.206406 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:59.206373    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532\": container with ID starting with a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532 not found: ID does not exist" containerID="a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"
Nov 04 00:31:59.206479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.206415    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532"} err="failed to get container status \"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532\": rpc error: code = NotFound desc = could not find container \"a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532\": container with ID starting with a63d966c7160027bf6b19ebf63f8f5c85cc1273d321a9550259af6b8c84c2532 not found: ID does not exist"
Nov 04 00:31:59.206479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.206434    6430 scope.go:122] "RemoveContainer" containerID="d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"
Nov 04 00:31:59.206731 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:31:59.206717    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc\": container with ID starting with d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc not found: ID does not exist" containerID="d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"
Nov 04 00:31:59.206786 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.206734    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc"} err="failed to get container status \"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc\": rpc error: code = NotFound desc = could not find container \"d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc\": container with ID starting with d47ed5b35dc35a3fa331ccb9fa2affb4d27b81b70ab6f9f1d56b02fc7a65febc not found: ID does not exist"
Nov 04 00:31:59.213944 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.213912    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 04 00:31:59.219709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:31:59.219681    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-330eb5-openshift-default-546d465f4c-hrt6n"]
Nov 04 00:32:00.232402 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:00.232361    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" path="/var/lib/kubelet/pods/25d3f943-66dd-4eea-a7c0-b6969c824c1b/volumes"
Nov 04 00:32:02.673613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:02.673574    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:32:02.674122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:02.673867    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy" containerID="cri-o://e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d" gracePeriod=30
Nov 04 00:32:04.833174 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:04.833129    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:32:07.977989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:07.977955    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:32:08.099589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099531    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.099589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099604    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.099847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099654    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.099847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099739    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.099847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099794    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.099847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099811    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.100070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099889    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.100070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.099949    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.100070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100010    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:08.100070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100050    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\") pod \"b4111f9a-5247-4406-889f-e5b99a2e9720\" (UID: \"b4111f9a-5247-4406-889f-e5b99a2e9720\") "
Nov 04 00:32:08.100298 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100072    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:08.100364 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100333    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:08.100529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100505    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:08.100626 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100522    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:08.100952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100926    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.100952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.100955    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.101192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.101025    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4111f9a-5247-4406-889f-e5b99a2e9720-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.101192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.101046    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.101192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.101062    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.102273 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.102239    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:08.102470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.102452    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:08.102554 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.102493    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:32:08.102606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.102558    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh" pod "b4111f9a-5247-4406-889f-e5b99a2e9720" (UID: "b4111f9a-5247-4406-889f-e5b99a2e9720"). InnerVolumeSpecName "kube-api-access-l2pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:08.202974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.202871    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.202974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.202916    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.202974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.202931    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.202974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.202946    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2pbh\" (UniqueName: \"kubernetes.io/projected/b4111f9a-5247-4406-889f-e5b99a2e9720-kube-api-access-l2pbh\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:08.250098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.249963    6430 generic.go:356] "Generic (PLEG): container finished" podID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerID="e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d" exitCode=0
Nov 04 00:32:08.250194 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.250139    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerDied","Data":"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"}
Nov 04 00:32:08.250256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.250200    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd" event={"ID":"b4111f9a-5247-4406-889f-e5b99a2e9720","Type":"ContainerDied","Data":"f2a36ef1513af47f94cf211da889e99e39cdbf49a7fd1ee34ad6c937aa2a3e1d"}
Nov 04 00:32:08.250256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.250208    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"
Nov 04 00:32:08.250256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.250229    6430 scope.go:122] "RemoveContainer" containerID="e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"
Nov 04 00:32:08.261123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.261098    6430 scope.go:122] "RemoveContainer" containerID="0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"
Nov 04 00:32:08.271805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.271771    6430 scope.go:122] "RemoveContainer" containerID="e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"
Nov 04 00:32:08.272346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.272319    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:32:08.272568 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:08.272357    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d\": container with ID starting with e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d not found: ID does not exist" containerID="e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"
Nov 04 00:32:08.272568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.272450    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d"} err="failed to get container status \"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d\": rpc error: code = NotFound desc = could not find container \"e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d\": container with ID starting with e887420115d85a9bbd4df9c93eb579ea915caef6111648fe1ed133b3e008c93d not found: ID does not exist"
Nov 04 00:32:08.272568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.272478    6430 scope.go:122] "RemoveContainer" containerID="0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"
Nov 04 00:32:08.272835 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:08.272806    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a\": container with ID starting with 0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a not found: ID does not exist" containerID="0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"
Nov 04 00:32:08.272929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.272844    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a"} err="failed to get container status \"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a\": rpc error: code = NotFound desc = could not find container \"0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a\": container with ID starting with 0f8bf87830584f9fb396007e35bc56eae269bde00877cb059e4a5cce5dadb73a not found: ID does not exist"
Nov 04 00:32:08.275998 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:08.275974    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-71b2f3-openshift-default-cf69c9bbf-4lbdd"]
Nov 04 00:32:10.246032 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:10.245994    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" path="/var/lib/kubelet/pods/b4111f9a-5247-4406-889f-e5b99a2e9720/volumes"
Nov 04 00:32:13.217567 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.217532    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 04 00:32:13.218378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.217830    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" podUID="188d9733-cd39-43f3-a917-4022645d9d2e" containerName="maas-api" containerID="cri-o://d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c" gracePeriod=30
Nov 04 00:32:13.618521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.618492    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 04 00:32:13.707247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.707198    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\") pod \"188d9733-cd39-43f3-a917-4022645d9d2e\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") "
Nov 04 00:32:13.707485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.707309    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\") pod \"188d9733-cd39-43f3-a917-4022645d9d2e\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") "
Nov 04 00:32:13.707485 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.707472    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\") pod \"188d9733-cd39-43f3-a917-4022645d9d2e\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") "
Nov 04 00:32:13.707611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.707515    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\" (UniqueName: \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\") pod \"188d9733-cd39-43f3-a917-4022645d9d2e\" (UID: \"188d9733-cd39-43f3-a917-4022645d9d2e\") "
Nov 04 00:32:13.707780 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.707744    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca" pod "188d9733-cd39-43f3-a917-4022645d9d2e" (UID: "188d9733-cd39-43f3-a917-4022645d9d2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:13.708181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.708154    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/188d9733-cd39-43f3-a917-4022645d9d2e-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:13.709675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.709648    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs" pod "188d9733-cd39-43f3-a917-4022645d9d2e" (UID: "188d9733-cd39-43f3-a917-4022645d9d2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:13.709675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.709664    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls" pod "188d9733-cd39-43f3-a917-4022645d9d2e" (UID: "188d9733-cd39-43f3-a917-4022645d9d2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:13.709798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.709711    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh" pod "188d9733-cd39-43f3-a917-4022645d9d2e" (UID: "188d9733-cd39-43f3-a917-4022645d9d2e"). InnerVolumeSpecName "kube-api-access-ntsnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:13.809937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.809902    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:13.809937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.809934    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/188d9733-cd39-43f3-a917-4022645d9d2e-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:13.809937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:13.809944    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsnh\" (UniqueName: \"kubernetes.io/projected/188d9733-cd39-43f3-a917-4022645d9d2e-kube-api-access-ntsnh\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:14.291013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.290972    6430 generic.go:356] "Generic (PLEG): container finished" podID="188d9733-cd39-43f3-a917-4022645d9d2e" containerID="d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c" exitCode=0
Nov 04 00:32:14.291502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.291027    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"
Nov 04 00:32:14.291502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.291045    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" event={"ID":"188d9733-cd39-43f3-a917-4022645d9d2e","Type":"ContainerDied","Data":"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"}
Nov 04 00:32:14.291502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.291090    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf" event={"ID":"188d9733-cd39-43f3-a917-4022645d9d2e","Type":"ContainerDied","Data":"4d97608a6b4801c153aaf82a9c506fae096a8debf69202abe9de4d54e57bbd2f"}
Nov 04 00:32:14.291502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.291117    6430 scope.go:122] "RemoveContainer" containerID="d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"
Nov 04 00:32:14.304799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.304776    6430 scope.go:122] "RemoveContainer" containerID="d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"
Nov 04 00:32:14.305079 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:14.305057    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c\": container with ID starting with d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c not found: ID does not exist" containerID="d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"
Nov 04 00:32:14.305127 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.305086    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c"} err="failed to get container status \"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c\": rpc error: code = NotFound desc = could not find container \"d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c\": container with ID starting with d19ae68786ac08036cc4f87ea08963ca550a4f5795f4f7cc25e19c9dc9be1d7c not found: ID does not exist"
Nov 04 00:32:14.311868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.311723    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 04 00:32:14.315568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:14.315543    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9a5948-86d685db84-pcbrf"]
Nov 04 00:32:16.232435 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:16.232402    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188d9733-cd39-43f3-a917-4022645d9d2e" path="/var/lib/kubelet/pods/188d9733-cd39-43f3-a917-4022645d9d2e/volumes"
Nov 04 00:32:18.496519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:18.496476    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 04 00:32:18.497056 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:18.496775    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy" containerID="cri-o://c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b" gracePeriod=30
Nov 04 00:32:23.694989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.694961    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 04 00:32:23.764527 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764486    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764560    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764611    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764684    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764784    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764800    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:23.764937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764865    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.764937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764913    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.765216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.764984    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\") pod \"e2e01cb9-1939-403b-9f99-467102f9417a\" (UID: \"e2e01cb9-1939-403b-9f99-467102f9417a\") "
Nov 04 00:32:23.765216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765116    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:32:23.765424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765271    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:23.765670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765645    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.765844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765711    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e01cb9-1939-403b-9f99-467102f9417a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.765844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765728    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.765844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765642    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:23.765844 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.765766    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:23.767492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.767468    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:23.767685 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.767657    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:32:23.767790 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.767702    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:23.767790 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.767748    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5" pod "e2e01cb9-1939-403b-9f99-467102f9417a" (UID: "e2e01cb9-1939-403b-9f99-467102f9417a"). InnerVolumeSpecName "kube-api-access-djsp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:23.866625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866590    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.866625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866620    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsp5\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-kube-api-access-djsp5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.866625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866631    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e01cb9-1939-403b-9f99-467102f9417a-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.866860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866640    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.866860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866649    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e01cb9-1939-403b-9f99-467102f9417a-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:23.866860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:23.866658    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e01cb9-1939-403b-9f99-467102f9417a-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:24.065327 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.065286    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 04 00:32:24.070473 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.070441    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 04 00:32:24.394526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.394434    6430 generic.go:356] "Generic (PLEG): container finished" podID="e2e01cb9-1939-403b-9f99-467102f9417a" containerID="c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b" exitCode=0
Nov 04 00:32:24.394526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.394496    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"
Nov 04 00:32:24.394794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.394520    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" event={"ID":"e2e01cb9-1939-403b-9f99-467102f9417a","Type":"ContainerDied","Data":"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"}
Nov 04 00:32:24.394794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.394587    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk" event={"ID":"e2e01cb9-1939-403b-9f99-467102f9417a","Type":"ContainerDied","Data":"4a1a59f4725970d357b258bddc22e9835d6744c12e02a8c7e472257085480e01"}
Nov 04 00:32:24.394794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.394618    6430 scope.go:122] "RemoveContainer" containerID="c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"
Nov 04 00:32:24.407806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.407786    6430 scope.go:122] "RemoveContainer" containerID="c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"
Nov 04 00:32:24.408090 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:24.408065    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b\": container with ID starting with c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b not found: ID does not exist" containerID="c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"
Nov 04 00:32:24.408163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.408104    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b"} err="failed to get container status \"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b\": rpc error: code = NotFound desc = could not find container \"c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b\": container with ID starting with c016215c5be3eca95a628fe2de7ea2362f8a55d55ec260ee07398fff12c9c26b not found: ID does not exist"
Nov 04 00:32:24.417804 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.417769    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 04 00:32:24.421853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:24.421830    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-d9xsk"]
Nov 04 00:32:26.235613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:26.235571    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" path="/var/lib/kubelet/pods/e2e01cb9-1939-403b-9f99-467102f9417a/volumes"
Nov 04 00:32:34.832161 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:34.832120    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:32:39.097905 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.097837    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main" containerID="cri-o://fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27" gracePeriod=45
Nov 04 00:32:39.098834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.098662    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" containerID="cri-o://946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5" gracePeriod=45
Nov 04 00:32:39.374006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.373977    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427476    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427538    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427571    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427634    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427725    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\" (UniqueName: \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.427805    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\") pod \"6b105d04-a066-4e09-92f5-99511832970a\" (UID: \"6b105d04-a066-4e09-92f5-99511832970a\") "
Nov 04 00:32:39.428909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.428570    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.428909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.428760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.429964 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.429931    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.430903 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.430875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.432054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.432029    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:39.439858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.439696    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j" pod "6b105d04-a066-4e09-92f5-99511832970a" (UID: "6b105d04-a066-4e09-92f5-99511832970a"). InnerVolumeSpecName "kube-api-access-kd69j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:39.443981 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.443951    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 04 00:32:39.522858 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.522826    6430 generic.go:356] "Generic (PLEG): container finished" podID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerID="946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5" exitCode=0
Nov 04 00:32:39.523059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.522866    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"
Nov 04 00:32:39.523059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.522904    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" event={"ID":"bc6f5e45-84ad-46f4-821d-75e9916977d5","Type":"ContainerDied","Data":"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"}
Nov 04 00:32:39.523059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.522947    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" event={"ID":"bc6f5e45-84ad-46f4-821d-75e9916977d5","Type":"ContainerDied","Data":"d0628c7904e8ea4102bf4d480715c8e1b03024e0a6c3bc2a1d0aa21c395bd74e"}
Nov 04 00:32:39.523059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.522958    6430 scope.go:122] "RemoveContainer" containerID="946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"
Nov 04 00:32:39.524501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.524464    6430 generic.go:356] "Generic (PLEG): container finished" podID="6b105d04-a066-4e09-92f5-99511832970a" containerID="fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27" exitCode=0
Nov 04 00:32:39.524623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.524503    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"
Nov 04 00:32:39.524623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.524515    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" event={"ID":"6b105d04-a066-4e09-92f5-99511832970a","Type":"ContainerDied","Data":"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"}
Nov 04 00:32:39.524623 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.524542    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz" event={"ID":"6b105d04-a066-4e09-92f5-99511832970a","Type":"ContainerDied","Data":"fae509db496ed7f0d23338f1331c7d4d9ec1284cb47db2d6e20993b461f34fc5"}
Nov 04 00:32:39.528784 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528763    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.528886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528829    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\" (UniqueName: \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.528886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528864    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.528986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528889    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.528986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.528986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.528970    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\") pod \"bc6f5e45-84ad-46f4-821d-75e9916977d5\" (UID: \"bc6f5e45-84ad-46f4-821d-75e9916977d5\") "
Nov 04 00:32:39.529184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529164    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.529242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529172    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.529348 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529322    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.529661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529637    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529665    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529678    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b105d04-a066-4e09-92f5-99511832970a-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529692    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529706    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529719    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.529747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529734    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.530098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529749    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd69j\" (UniqueName: \"kubernetes.io/projected/6b105d04-a066-4e09-92f5-99511832970a-kube-api-access-kd69j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.530098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.529764    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b105d04-a066-4e09-92f5-99511832970a-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.531003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.530977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:32:39.531282 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.531262    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:32:39.531883 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.531864    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr" pod "bc6f5e45-84ad-46f4-821d-75e9916977d5" (UID: "bc6f5e45-84ad-46f4-821d-75e9916977d5"). InnerVolumeSpecName "kube-api-access-p5zdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:32:39.533714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.533545    6430 scope.go:122] "RemoveContainer" containerID="946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"
Nov 04 00:32:39.534014 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:39.533984    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5\": container with ID starting with 946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5 not found: ID does not exist" containerID="946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"
Nov 04 00:32:39.534110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.534022    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5"} err="failed to get container status \"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5\": rpc error: code = NotFound desc = could not find container \"946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5\": container with ID starting with 946d9518087b667b69e6b5d12a5d29b1483050a45602b5bbecbd5fa72c1ecbb5 not found: ID does not exist"
Nov 04 00:32:39.534110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.534044    6430 scope.go:122] "RemoveContainer" containerID="fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"
Nov 04 00:32:39.542617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.542578    6430 scope.go:122] "RemoveContainer" containerID="fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"
Nov 04 00:32:39.542824 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:32:39.542803    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27\": container with ID starting with fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27 not found: ID does not exist" containerID="fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"
Nov 04 00:32:39.542889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.542829    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27"} err="failed to get container status \"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27\": rpc error: code = NotFound desc = could not find container \"fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27\": container with ID starting with fd98f9159b23c1630f62bc1d95566bbaf9b70176c64a642de79d1c19e0dabd27 not found: ID does not exist"
Nov 04 00:32:39.549902 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.549877    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 04 00:32:39.554328 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.554306    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-2-w0-f6f292-kserve-58899dc54f-rwsqz"]
Nov 04 00:32:39.630884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.630801    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6f5e45-84ad-46f4-821d-75e9916977d5-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.630884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.630832    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zdr\" (UniqueName: \"kubernetes.io/projected/bc6f5e45-84ad-46f4-821d-75e9916977d5-kube-api-access-p5zdr\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.630884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.630842    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6f5e45-84ad-46f4-821d-75e9916977d5-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:32:39.847962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.847927    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 04 00:32:39.851869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:39.851842    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2"]
Nov 04 00:32:40.232625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:40.232590    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b105d04-a066-4e09-92f5-99511832970a" path="/var/lib/kubelet/pods/6b105d04-a066-4e09-92f5-99511832970a/volumes"
Nov 04 00:32:40.233089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:40.233074    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" path="/var/lib/kubelet/pods/bc6f5e45-84ad-46f4-821d-75e9916977d5/volumes"
Nov 04 00:32:40.433216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:40.433123    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/distinct-w0-f6f292-kserve-7f5865cfd6-hb2b2" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:32:46.120798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:32:46.120752    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:33:04.837572 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:33:04.837526    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:33:05.053415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:05.048186    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 04 00:33:05.053415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:05.048477    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" containerID="cri-o://48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9" gracePeriod=30
Nov 04 00:33:07.646289 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:07.646242    6430 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:33:07.646834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:07.646326    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:33:10.327462 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.327382    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 04 00:33:10.442613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442583    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.442831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442651    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.442831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442729    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.442959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442833    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.442959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442890    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.442959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442942    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.443098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442978    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.443098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.442999    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:10.443178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443096    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:10.443246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443201    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:10.443246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443206    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.443351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443183    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:10.443351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443276    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\") pod \"e24df8d1-00b8-4c1f-a891-d5110cec430f\" (UID: \"e24df8d1-00b8-4c1f-a891-d5110cec430f\") "
Nov 04 00:33:10.443578 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443553    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:10.443987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443960    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.444093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.443991    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.444093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.444010    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.444093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.444026    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24df8d1-00b8-4c1f-a891-d5110cec430f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.444093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.444041    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.445199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.445161    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:10.445279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.445205    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "kube-api-access-9jjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:10.445279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.445212    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:10.445279 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.445248    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo" pod "e24df8d1-00b8-4c1f-a891-d5110cec430f" (UID: "e24df8d1-00b8-4c1f-a891-d5110cec430f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:33:10.545451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.545410    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jjdl\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-kube-api-access-9jjdl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.545451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.545451    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.545669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.545466    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.545669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.545481    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24df8d1-00b8-4c1f-a891-d5110cec430f-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:10.750663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.750581    6430 generic.go:356] "Generic (PLEG): container finished" podID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerID="48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9" exitCode=0
Nov 04 00:33:10.750663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.750632    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"
Nov 04 00:33:10.750867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.750663    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" event={"ID":"e24df8d1-00b8-4c1f-a891-d5110cec430f","Type":"ContainerDied","Data":"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"}
Nov 04 00:33:10.750867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.750725    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x" event={"ID":"e24df8d1-00b8-4c1f-a891-d5110cec430f","Type":"ContainerDied","Data":"cc1afdfd4ac6eaeaaacb22356f6c31397d8d1b9036b43c262441b81615786777"}
Nov 04 00:33:10.750867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.750754    6430 scope.go:122] "RemoveContainer" containerID="48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"
Nov 04 00:33:10.761705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.761683    6430 scope.go:122] "RemoveContainer" containerID="48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"
Nov 04 00:33:10.761971 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:33:10.761950    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9\": container with ID starting with 48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9 not found: ID does not exist" containerID="48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"
Nov 04 00:33:10.762045 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.761981    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9"} err="failed to get container status \"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9\": rpc error: code = NotFound desc = could not find container \"48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9\": container with ID starting with 48171489d4bf444d66754b7d4b66ec6a93641e519fdf16e8b69c23d1ee5107b9 not found: ID does not exist"
Nov 04 00:33:10.776323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.776294    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 04 00:33:10.781658 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:10.781637    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-l4g6x"]
Nov 04 00:33:12.232598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.232566    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" path="/var/lib/kubelet/pods/e24df8d1-00b8-4c1f-a891-d5110cec430f/volumes"
Nov 04 00:33:12.586986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.586946    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:33:12.587611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587594    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587612    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587629    6430 cpu_manager.go:414] "containerMap: removing container" podUID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" containerName="maas-api"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587634    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" containerName="maas-api"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587640    6430 cpu_manager.go:414] "containerMap: removing container" podUID="188d9733-cd39-43f3-a917-4022645d9d2e" containerName="maas-api"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587648    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188d9733-cd39-43f3-a917-4022645d9d2e" containerName="maas-api"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587656    6430 cpu_manager.go:414] "containerMap: removing container" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587661    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587669    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.587677 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587673    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587685    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587689    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587700    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587705    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587713    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587718    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587726    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587730    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587739    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587744    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587751    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587755    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587760    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" containerName="payload-pre-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587765    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" containerName="payload-pre-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587773    6430 cpu_manager.go:414] "containerMap: removing container" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587777    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587785    6430 cpu_manager.go:414] "containerMap: removing container" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587791    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587798    6430 cpu_manager.go:414] "containerMap: removing container" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587807    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587812    6430 cpu_manager.go:414] "containerMap: removing container" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587817    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587825    6430 cpu_manager.go:414] "containerMap: removing container" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587829    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587835    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587840    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587847    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" containerName="payload-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587851    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" containerName="payload-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587866    6430 cpu_manager.go:414] "containerMap: removing container" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587870    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587878    6430 cpu_manager.go:414] "containerMap: removing container" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587882    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587889    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" containerName="payload-pre-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587894    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" containerName="payload-pre-processing"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587903    6430 cpu_manager.go:414] "containerMap: removing container" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587907    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587914    6430 cpu_manager.go:414] "containerMap: removing container" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587918    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587926    6430 cpu_manager.go:414] "containerMap: removing container" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587930    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587941    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.587946    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:12.588006 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588021    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588029    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588036    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588043    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24df8d1-00b8-4c1f-a891-d5110cec430f" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588050    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588055    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588060    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588067    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc82e6b-c5d9-4863-8712-70ebc2236f8c" containerName="maas-api"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588071    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="188d9733-cd39-43f3-a917-4022645d9d2e" containerName="maas-api"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588076    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588082    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588088    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b105d04-a066-4e09-92f5-99511832970a" containerName="main"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588094    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4111f9a-5247-4406-889f-e5b99a2e9720" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588101    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588106    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8cbd67-a8c5-4333-b72c-e94997ded54a" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588110    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb20907-450f-42e1-bfc6-c560ea4ab7fd" containerName="payload-pre-processing"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588115    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588120    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a79208-d133-4c26-90dd-364f5146ee8a" containerName="maas-api"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588125    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588131    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d3f943-66dd-4eea-a7c0-b6969c824c1b" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588137    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e01cb9-1939-403b-9f99-467102f9417a" containerName="istio-proxy"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588144    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9561d9a3-02b4-43a6-a1a7-4cab5c54a311" containerName="payload-processing"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588150    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588156    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6f5e45-84ad-46f4-821d-75e9916977d5" containerName="main"
Nov 04 00:33:12.589492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.588160    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4991b2-f3c9-445f-b4ec-4d8a7433acb2" containerName="payload-pre-processing"
Nov 04 00:33:12.593343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.593324    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.603128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.603103    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-b3bea7-openshift-default-dockercfg-x9m4w\""
Nov 04 00:33:12.605197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.605167    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:33:12.673101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673062    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673175    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673212    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673254    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673357    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673378    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673426    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673594    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.673734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.673687    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxbm\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775306    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxbm\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775376    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775440    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775456    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775489    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775598    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775633    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.775809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775744    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.776039 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775910    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.776039 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.775957    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.776153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.776039    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.776153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.776116    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.776420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.776378    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.778009 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.777989    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.778184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.778165    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.788068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.788040    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.788197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.788175    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxbm\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:12.910684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:12.910588    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:13.201530 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.201500    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:33:13.203775 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:33:13.203721    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3508bcc2_87c3_4f2a_b2da_8aa6c4192299.slice/crio-d759ac527592a8c6a9c708ba51a4e4b59a68dd9ddb844a90d13bf9abb8671c39 WatchSource:0}: Error finding container d759ac527592a8c6a9c708ba51a4e4b59a68dd9ddb844a90d13bf9abb8671c39: Status 404 returned error can't find the container with id d759ac527592a8c6a9c708ba51a4e4b59a68dd9ddb844a90d13bf9abb8671c39
Nov 04 00:33:13.207180 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.207143    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:13.207272 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.207229    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:13.207320 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.207271    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:13.720744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.720699    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 04 00:33:13.721206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.720948    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" podUID="5118d76b-260a-4cc7-9965-90df51e80bc9" containerName="maas-api" containerID="cri-o://55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e" gracePeriod=30
Nov 04 00:33:13.772856 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.772823    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" event={"ID":"3508bcc2-87c3-4f2a-b2da-8aa6c4192299","Type":"ContainerStarted","Data":"3fcdb26b471397c0ff409ca148b5b08b6a4d0e7aa6da1d2223e2fbbf300d4f39"}
Nov 04 00:33:13.773009 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.772886    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" event={"ID":"3508bcc2-87c3-4f2a-b2da-8aa6c4192299","Type":"ContainerStarted","Data":"d759ac527592a8c6a9c708ba51a4e4b59a68dd9ddb844a90d13bf9abb8671c39"}
Nov 04 00:33:13.798271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.798224    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podStartSLOduration=1.798212576 podStartE2EDuration="1.798212576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:33:13.798036883 +0000 UTC m=+3809.689098820" watchObservedRunningTime="2026-11-04 00:33:13.798212576 +0000 UTC m=+3809.689274510"
Nov 04 00:33:13.911668 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:13.911634    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:14.071219 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.071189    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 04 00:33:14.221672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.221636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\" (UniqueName: \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\") pod \"5118d76b-260a-4cc7-9965-90df51e80bc9\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") "
Nov 04 00:33:14.221987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.221969    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\") pod \"5118d76b-260a-4cc7-9965-90df51e80bc9\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") "
Nov 04 00:33:14.225494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.225462    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\") pod \"5118d76b-260a-4cc7-9965-90df51e80bc9\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") "
Nov 04 00:33:14.226780 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.226572    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\") pod \"5118d76b-260a-4cc7-9965-90df51e80bc9\" (UID: \"5118d76b-260a-4cc7-9965-90df51e80bc9\") "
Nov 04 00:33:14.230005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.229931    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca" pod "5118d76b-260a-4cc7-9965-90df51e80bc9" (UID: "5118d76b-260a-4cc7-9965-90df51e80bc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:33:14.237150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.237121    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs" pod "5118d76b-260a-4cc7-9965-90df51e80bc9" (UID: "5118d76b-260a-4cc7-9965-90df51e80bc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:14.240896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.240855    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l" pod "5118d76b-260a-4cc7-9965-90df51e80bc9" (UID: "5118d76b-260a-4cc7-9965-90df51e80bc9"). InnerVolumeSpecName "kube-api-access-2gn8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:14.242788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.242758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls" pod "5118d76b-260a-4cc7-9965-90df51e80bc9" (UID: "5118d76b-260a-4cc7-9965-90df51e80bc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:14.329130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.329098    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gn8l\" (UniqueName: \"kubernetes.io/projected/5118d76b-260a-4cc7-9965-90df51e80bc9-kube-api-access-2gn8l\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:14.329130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.329127    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5118d76b-260a-4cc7-9965-90df51e80bc9-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:14.329130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.329137    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:14.329130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.329146    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5118d76b-260a-4cc7-9965-90df51e80bc9-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:14.779822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.779733    6430 generic.go:356] "Generic (PLEG): container finished" podID="5118d76b-260a-4cc7-9965-90df51e80bc9" containerID="55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e" exitCode=0
Nov 04 00:33:14.779822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.779777    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"
Nov 04 00:33:14.780354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.779816    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" event={"ID":"5118d76b-260a-4cc7-9965-90df51e80bc9","Type":"ContainerDied","Data":"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"}
Nov 04 00:33:14.780354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.779885    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b" event={"ID":"5118d76b-260a-4cc7-9965-90df51e80bc9","Type":"ContainerDied","Data":"91a27ab18b9474fe0d97ae3fd2bc295d136ed58bb9295b4a7e53073475f94ec1"}
Nov 04 00:33:14.780354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.779934    6430 scope.go:122] "RemoveContainer" containerID="55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"
Nov 04 00:33:14.798501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.797216    6430 scope.go:122] "RemoveContainer" containerID="55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"
Nov 04 00:33:14.798501 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:33:14.798430    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e\": container with ID starting with 55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e not found: ID does not exist" containerID="55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"
Nov 04 00:33:14.798501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.798465    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e"} err="failed to get container status \"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e\": rpc error: code = NotFound desc = could not find container \"55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e\": container with ID starting with 55c3b6ccdb7352bd913f2f1e3b1588ce7fde0275d997d8bf3287300b9cac875e not found: ID does not exist"
Nov 04 00:33:14.830412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.827470    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 04 00:33:14.834136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.834094    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2b1f66-7c87547d6-n748b"]
Nov 04 00:33:14.932924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.932883    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:14.933139 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:14.932944    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:15.920768 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:15.920726    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:15.921362 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:15.921323    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:16.241577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:16.241480    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5118d76b-260a-4cc7-9965-90df51e80bc9" path="/var/lib/kubelet/pods/5118d76b-260a-4cc7-9965-90df51e80bc9/volumes"
Nov 04 00:33:16.928408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:16.922912    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:16.928408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:16.922977    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:17.926413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:17.924820    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:17.926413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:17.924897    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:18.932544 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:18.932497    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:18.932950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:18.932562    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:19.929413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:19.928814    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 04 00:33:19.929413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:19.928921    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 04 00:33:20.916482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:20.916446    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:20.916919 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:20.916896    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:20.917673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:20.917653    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:33:21.101048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:21.100989    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 04 00:33:28.498723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.498684    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:33:28.499276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499263    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:28.499324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499276    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:28.499324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499293    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5118d76b-260a-4cc7-9965-90df51e80bc9" containerName="maas-api"
Nov 04 00:33:28.499324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499298    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5118d76b-260a-4cc7-9965-90df51e80bc9" containerName="maas-api"
Nov 04 00:33:28.499475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499332    6430 cpu_manager.go:414] "containerMap: removing container" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:28.499475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499339    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a3a9c0-8259-49b0-848d-bac8c548a99c" containerName="istio-proxy"
Nov 04 00:33:28.499475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499454    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5118d76b-260a-4cc7-9965-90df51e80bc9" containerName="maas-api"
Nov 04 00:33:28.499475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499465    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e3984c-0695-4a78-92ef-cca9fdab2999" containerName="cleanup"
Nov 04 00:33:28.499475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499471    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b81506-8e1c-4c9d-8067-f90d0c26f920" containerName="cleanup"
Nov 04 00:33:28.499641 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.499480    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c76493c-8efd-42a4-90d6-cefa92b7788e" containerName="istio-proxy"
Nov 04 00:33:28.502617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.502592    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.507580 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.507557    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-15b02a-openshift-default-dockercfg-nsn9f\""
Nov 04 00:33:28.517070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.517045    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:33:28.580732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.580695    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.580909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.580767    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.580909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.580815    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlb4w\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.580909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.580896    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.581026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.580930    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.581026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.581011    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.581095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.581039    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.581095 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.581062    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.581223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.581201    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682271 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682239    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682294    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682340    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682358    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlb4w\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682461    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682526    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682615    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682661    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682697    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682822    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.682970 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.682906    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.683093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.683003    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.683161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.683111    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.683512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.683488    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.684999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.684975    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.685403 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.685361    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.695278 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.695255    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlb4w\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.695365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.695307    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:28.820469 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:28.820435    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:29.107884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.107843    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:33:29.109272 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:33:29.109245    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73298ba3_d100_4e00_8708_cc9de5588a9e.slice/crio-2eae1ace97a8d19113bbd5e21fb56761c732b7ce7f78027fd9ef0d70a5c6d3c9 WatchSource:0}: Error finding container 2eae1ace97a8d19113bbd5e21fb56761c732b7ce7f78027fd9ef0d70a5c6d3c9: Status 404 returned error can't find the container with id 2eae1ace97a8d19113bbd5e21fb56761c732b7ce7f78027fd9ef0d70a5c6d3c9
Nov 04 00:33:29.111930 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.111895    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:29.112016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.111963    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:29.112016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.111992    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:33:29.980910 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.980874    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" event={"ID":"73298ba3-d100-4e00-8708-cc9de5588a9e","Type":"ContainerStarted","Data":"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"}
Nov 04 00:33:29.981333 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:29.980930    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" event={"ID":"73298ba3-d100-4e00-8708-cc9de5588a9e","Type":"ContainerStarted","Data":"2eae1ace97a8d19113bbd5e21fb56761c732b7ce7f78027fd9ef0d70a5c6d3c9"}
Nov 04 00:33:30.032318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:30.032256    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podStartSLOduration=2.032240809 podStartE2EDuration="2.032240809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:33:30.031819907 +0000 UTC m=+3825.922881835" watchObservedRunningTime="2026-11-04 00:33:30.032240809 +0000 UTC m=+3825.923302743"
Nov 04 00:33:30.831576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:30.831526    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:30.831800 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:30.831595    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 04 00:33:30.831800 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:30.831643    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 04 00:33:31.830411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:31.827840    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 04 00:33:31.830411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:31.827934    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 04 00:33:32.831911 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:32.831858    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 04 00:33:32.832445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:32.831929    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 04 00:33:33.826411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:33.822923    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 04 00:33:33.826411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:33.822986    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 04 00:33:34.833305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:34.833254    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 04 00:33:34.833798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:34.833353    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 04 00:33:34.857770 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:33:34.857733    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:33:35.846367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:35.846326    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:35.847339 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:35.847298    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:35.848944 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:35.848911    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:33:36.128566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.128464    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main" containerID="cri-o://c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307" gracePeriod=45
Nov 04 00:33:36.410525 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.410497    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 04 00:33:36.529494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529417    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.529691 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529666    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.529779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529695    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.529779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529715    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.529910 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529838    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.529910 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529883    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\" (UniqueName: \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.530025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.529951    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\") pod \"689807c7-203c-4dec-82fa-a2a8925be4ae\" (UID: \"689807c7-203c-4dec-82fa-a2a8925be4ae\") "
Nov 04 00:33:36.530025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.530005    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.530281 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.530256    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.530581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.530557    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.530696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.530586    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.530696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.530600    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.532197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.532173    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:33:36.532314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.532266    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "kube-api-access-mxfpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:33:36.532381 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.532306    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm" pod "689807c7-203c-4dec-82fa-a2a8925be4ae" (UID: "689807c7-203c-4dec-82fa-a2a8925be4ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:33:36.632640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.632556    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/689807c7-203c-4dec-82fa-a2a8925be4ae-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.632640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.632587    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfpn\" (UniqueName: \"kubernetes.io/projected/689807c7-203c-4dec-82fa-a2a8925be4ae-kube-api-access-mxfpn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:36.632640 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:36.632598    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/689807c7-203c-4dec-82fa-a2a8925be4ae-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:33:37.048432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.048400    6430 generic.go:356] "Generic (PLEG): container finished" podID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerID="c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307" exitCode=0
Nov 04 00:33:37.048924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.048444    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"
Nov 04 00:33:37.048924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.048483    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" event={"ID":"689807c7-203c-4dec-82fa-a2a8925be4ae","Type":"ContainerDied","Data":"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"}
Nov 04 00:33:37.048924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.048522    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk" event={"ID":"689807c7-203c-4dec-82fa-a2a8925be4ae","Type":"ContainerDied","Data":"e99e6f34f66c5dc87f9dba4e1afd4f67578b66ecc5704033a676424e24db44d4"}
Nov 04 00:33:37.048924 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.048533    6430 scope.go:122] "RemoveContainer" containerID="c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"
Nov 04 00:33:37.059445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.059420    6430 scope.go:122] "RemoveContainer" containerID="c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"
Nov 04 00:33:37.060040 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:33:37.059857    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307\": container with ID starting with c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307 not found: ID does not exist" containerID="c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"
Nov 04 00:33:37.060040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.059896    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307"} err="failed to get container status \"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307\": rpc error: code = NotFound desc = could not find container \"c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307\": container with ID starting with c44a64eab5b0d493bc166477ca548fe943992dc90c0791749e5cef8f1467c307 not found: ID does not exist"
Nov 04 00:33:37.075942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.075916    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 04 00:33:37.080341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:37.080320    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/simulator-w1-5cb349-kserve-69954974f6-9f8lk"]
Nov 04 00:33:38.231949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:38.231911    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" path="/var/lib/kubelet/pods/689807c7-203c-4dec-82fa-a2a8925be4ae/volumes"
Nov 04 00:33:46.951980 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:33:46.951945    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:34:04.831331 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:34:04.831296    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:34:17.215888 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:34:17.215855    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:34:34.831267 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:34:34.831231    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:34:44.204319 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:34:44.204276    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:35:04.832227 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:04.832179    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:35:05.166691 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:05.166613    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:35:05.166855 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:05.166828    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy" containerID="cri-o://de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e" gracePeriod=30
Nov 04 00:35:10.430128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.430096    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:35:10.552276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552248    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552375    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552408    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552438    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552495    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552546    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552555    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:10.552676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552576    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552603    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552685    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\") pod \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\" (UID: \"a25d18f2-34e3-4912-8f9b-0cffcaec49da\") "
Nov 04 00:35:10.552973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552749    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:10.552973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.552879    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:10.553166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553143    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:10.553229 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553125    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:10.553367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553344    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.553492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553372    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.553492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553421    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.553492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553437    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.553492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.553451    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.554792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.554768    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:10.554792 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.554778    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:10.554949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.554842    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:10.554949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.554895    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6" pod "a25d18f2-34e3-4912-8f9b-0cffcaec49da" (UID: "a25d18f2-34e3-4912-8f9b-0cffcaec49da"). InnerVolumeSpecName "kube-api-access-g2jz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:10.655151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.655117    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.655151 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.655151    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.655365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.655164    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.655365 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.655180    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jz6\" (UniqueName: \"kubernetes.io/projected/a25d18f2-34e3-4912-8f9b-0cffcaec49da-kube-api-access-g2jz6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:10.710461 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.710430    6430 generic.go:356] "Generic (PLEG): container finished" podID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerID="de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e" exitCode=0
Nov 04 00:35:10.710619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.710512    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"
Nov 04 00:35:10.710619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.710519    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerDied","Data":"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"}
Nov 04 00:35:10.710619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.710565    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn" event={"ID":"a25d18f2-34e3-4912-8f9b-0cffcaec49da","Type":"ContainerDied","Data":"09682d3ba6256b2b6ea4ec366ba7f4de2a94d053f1ba9ba090330a1b7260b163"}
Nov 04 00:35:10.710619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.710589    6430 scope.go:122] "RemoveContainer" containerID="de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"
Nov 04 00:35:10.721521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.721501    6430 scope.go:122] "RemoveContainer" containerID="e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"
Nov 04 00:35:10.730773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.730755    6430 scope.go:122] "RemoveContainer" containerID="de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"
Nov 04 00:35:10.731014 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:10.730998    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e\": container with ID starting with de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e not found: ID does not exist" containerID="de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"
Nov 04 00:35:10.731063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.731023    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e"} err="failed to get container status \"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e\": rpc error: code = NotFound desc = could not find container \"de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e\": container with ID starting with de02944fd91b9fefe7ad0389460344396cedcb161e8ecbc1b3fb99d3d867c91e not found: ID does not exist"
Nov 04 00:35:10.731063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.731038    6430 scope.go:122] "RemoveContainer" containerID="e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"
Nov 04 00:35:10.731257 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:10.731238    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd\": container with ID starting with e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd not found: ID does not exist" containerID="e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"
Nov 04 00:35:10.731297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.731262    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd"} err="failed to get container status \"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd\": rpc error: code = NotFound desc = could not find container \"e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd\": container with ID starting with e5b8dc0aa9b4c9c7b6e6adcbfdbb3173e5e43d432411024489b5c488546a48dd not found: ID does not exist"
Nov 04 00:35:10.736115 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.736090    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:35:10.740334 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.740313    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-86deca-openshift-default-686c55f8fc-48tmn"]
Nov 04 00:35:10.816584 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.816509    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:35:10.816753 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:10.816729    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" podUID="1d51fc39-822d-4d36-af2f-5dd386f57a78" containerName="maas-api" containerID="cri-o://694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6" gracePeriod=30
Nov 04 00:35:11.117048 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.117015    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:35:11.265877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.265837    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\") pod \"1d51fc39-822d-4d36-af2f-5dd386f57a78\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") "
Nov 04 00:35:11.268167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.266609    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\" (UniqueName: \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\") pod \"1d51fc39-822d-4d36-af2f-5dd386f57a78\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") "
Nov 04 00:35:11.268167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.267265    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\") pod \"1d51fc39-822d-4d36-af2f-5dd386f57a78\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") "
Nov 04 00:35:11.268167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.267962    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\") pod \"1d51fc39-822d-4d36-af2f-5dd386f57a78\" (UID: \"1d51fc39-822d-4d36-af2f-5dd386f57a78\") "
Nov 04 00:35:11.270896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.269007    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca" pod "1d51fc39-822d-4d36-af2f-5dd386f57a78" (UID: "1d51fc39-822d-4d36-af2f-5dd386f57a78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:11.270896 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.269099    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs" pod "1d51fc39-822d-4d36-af2f-5dd386f57a78" (UID: "1d51fc39-822d-4d36-af2f-5dd386f57a78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:11.279105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.278134    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79" pod "1d51fc39-822d-4d36-af2f-5dd386f57a78" (UID: "1d51fc39-822d-4d36-af2f-5dd386f57a78"). InnerVolumeSpecName "kube-api-access-9gd79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:11.279105 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.278236    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls" pod "1d51fc39-822d-4d36-af2f-5dd386f57a78" (UID: "1d51fc39-822d-4d36-af2f-5dd386f57a78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:11.369995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.369883    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d51fc39-822d-4d36-af2f-5dd386f57a78-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:11.369995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.369922    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:11.369995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.369937    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gd79\" (UniqueName: \"kubernetes.io/projected/1d51fc39-822d-4d36-af2f-5dd386f57a78-kube-api-access-9gd79\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:11.369995 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.369952    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d51fc39-822d-4d36-af2f-5dd386f57a78-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:11.718705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.718612    6430 generic.go:356] "Generic (PLEG): container finished" podID="1d51fc39-822d-4d36-af2f-5dd386f57a78" containerID="694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6" exitCode=0
Nov 04 00:35:11.718705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.718669    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"
Nov 04 00:35:11.718705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.718696    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" event={"ID":"1d51fc39-822d-4d36-af2f-5dd386f57a78","Type":"ContainerDied","Data":"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"}
Nov 04 00:35:11.719341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.718750    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt" event={"ID":"1d51fc39-822d-4d36-af2f-5dd386f57a78","Type":"ContainerDied","Data":"d1de392566b11c61fd487fe0be8c74296cba71cc9aea66fba1b3011834f6dd0f"}
Nov 04 00:35:11.719341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.718773    6430 scope.go:122] "RemoveContainer" containerID="694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"
Nov 04 00:35:11.729624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.729606    6430 scope.go:122] "RemoveContainer" containerID="694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"
Nov 04 00:35:11.729877 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:11.729860    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6\": container with ID starting with 694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6 not found: ID does not exist" containerID="694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"
Nov 04 00:35:11.729950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.729888    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6"} err="failed to get container status \"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6\": rpc error: code = NotFound desc = could not find container \"694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6\": container with ID starting with 694ec866c871d874db4f152dcc57d51019d83d819f44770b2419bc353ebb4ca6 not found: ID does not exist"
Nov 04 00:35:11.745126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.745097    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:35:11.747529 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:11.747506    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fa0d0a-6b59749d9-z9ttt"]
Nov 04 00:35:12.290409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:12.287179    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d51fc39-822d-4d36-af2f-5dd386f57a78" path="/var/lib/kubelet/pods/1d51fc39-822d-4d36-af2f-5dd386f57a78/volumes"
Nov 04 00:35:12.322441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:12.311370    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" path="/var/lib/kubelet/pods/a25d18f2-34e3-4912-8f9b-0cffcaec49da/volumes"
Nov 04 00:35:22.414556 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:22.414520    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:35:28.597824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.594819    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:35:28.597824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.595115    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy" containerID="cri-o://a5810fb9434db5941bc68d70b3143e47694b8651e0967c1ab7d212c398c1027a" gracePeriod=30
Nov 04 00:35:28.604601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.603593    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:35:28.604601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.603849    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy" containerID="cri-o://b49c76bf77a085fd930b5ec31ac6c10d6d177ab37deb1d881a70d5886058c658" gracePeriod=30
Nov 04 00:35:28.610506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.610473    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 04 00:35:28.610749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.610703    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy" containerID="cri-o://f1aca37da1818d8a06da9f5f826890d9a0dbdbe315df15695e2be80ff2ccbfc9" gracePeriod=30
Nov 04 00:35:28.611812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.611783    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:35:28.611972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.611952    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy" containerID="cri-o://6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260" gracePeriod=30
Nov 04 00:35:28.760206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.760171    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 04 00:35:28.766347 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:28.766309    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 04 00:35:29.300483 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.300445    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 04 00:35:29.300725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.300685    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" podUID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" containerName="maas-api" containerID="cri-o://a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf" gracePeriod=30
Nov 04 00:35:29.655868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.655839    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 04 00:35:29.731074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731037    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\") pod \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") "
Nov 04 00:35:29.731244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731161    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\" (UniqueName: \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\") pod \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") "
Nov 04 00:35:29.731244 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731212    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\") pod \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") "
Nov 04 00:35:29.731341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731314    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\") pod \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\" (UID: \"0c8600c0-1c62-4155-9bb0-4a13f50bb8df\") "
Nov 04 00:35:29.731410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731374    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca" pod "0c8600c0-1c62-4155-9bb0-4a13f50bb8df" (UID: "0c8600c0-1c62-4155-9bb0-4a13f50bb8df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:29.731810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.731787    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:29.733273 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.733248    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l" pod "0c8600c0-1c62-4155-9bb0-4a13f50bb8df" (UID: "0c8600c0-1c62-4155-9bb0-4a13f50bb8df"). InnerVolumeSpecName "kube-api-access-5z92l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:29.733372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.733328    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs" pod "0c8600c0-1c62-4155-9bb0-4a13f50bb8df" (UID: "0c8600c0-1c62-4155-9bb0-4a13f50bb8df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:29.733473 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.733454    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls" pod "0c8600c0-1c62-4155-9bb0-4a13f50bb8df" (UID: "0c8600c0-1c62-4155-9bb0-4a13f50bb8df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:29.841090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.839090    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z92l\" (UniqueName: \"kubernetes.io/projected/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-kube-api-access-5z92l\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:29.841090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.839150    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:29.841090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.839167    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8600c0-1c62-4155-9bb0-4a13f50bb8df-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:29.869537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.869502    6430 generic.go:356] "Generic (PLEG): container finished" podID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" containerID="a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf" exitCode=0
Nov 04 00:35:29.869739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.869548    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"
Nov 04 00:35:29.869739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.869587    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" event={"ID":"0c8600c0-1c62-4155-9bb0-4a13f50bb8df","Type":"ContainerDied","Data":"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"}
Nov 04 00:35:29.869739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.869647    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4" event={"ID":"0c8600c0-1c62-4155-9bb0-4a13f50bb8df","Type":"ContainerDied","Data":"12efb13116693f72966c7ddff6991c9052fce70ed8ed6afa5fecdffda0e89938"}
Nov 04 00:35:29.869739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.869679    6430 scope.go:122] "RemoveContainer" containerID="a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"
Nov 04 00:35:29.880429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.880376    6430 scope.go:122] "RemoveContainer" containerID="a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"
Nov 04 00:35:29.880724 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:29.880704    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf\": container with ID starting with a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf not found: ID does not exist" containerID="a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"
Nov 04 00:35:29.880772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.880731    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf"} err="failed to get container status \"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf\": rpc error: code = NotFound desc = could not find container \"a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf\": container with ID starting with a34f29e6fbd420bf531bf9c37dd6eaa9c4ebb7a19152b431c8d2faee2816c8bf not found: ID does not exist"
Nov 04 00:35:29.897130 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.897109    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 04 00:35:29.899310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:29.899291    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b75d-cccc5d868-wqlr4"]
Nov 04 00:35:30.232454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:30.232347    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" path="/var/lib/kubelet/pods/0c8600c0-1c62-4155-9bb0-4a13f50bb8df/volumes"
Nov 04 00:35:33.761211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.761149    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing" containerID="cri-o://e2f546c75723787bd12243f5d09a173030eef39e9ba7261f0e8b878767a3d213" gracePeriod=25
Nov 04 00:35:33.769370 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.769314    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing" containerID="cri-o://02b81c304946e46f828e98c878698b79c9100e7aaeb2a8c1c57d06ca200fb5b7" gracePeriod=25
Nov 04 00:35:33.891445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.891328    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:35:33.901658 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.901491    6430 generic.go:356] "Generic (PLEG): container finished" podID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerID="f1aca37da1818d8a06da9f5f826890d9a0dbdbe315df15695e2be80ff2ccbfc9" exitCode=0
Nov 04 00:35:33.901658 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.901561    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" event={"ID":"dcda01ba-ca83-4671-8e3c-b2ffc559f992","Type":"ContainerDied","Data":"f1aca37da1818d8a06da9f5f826890d9a0dbdbe315df15695e2be80ff2ccbfc9"}
Nov 04 00:35:33.904494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.904464    6430 generic.go:356] "Generic (PLEG): container finished" podID="1226101c-28de-4eed-a854-c6db2a7b8540" containerID="b49c76bf77a085fd930b5ec31ac6c10d6d177ab37deb1d881a70d5886058c658" exitCode=0
Nov 04 00:35:33.904635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.904555    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" event={"ID":"1226101c-28de-4eed-a854-c6db2a7b8540","Type":"ContainerDied","Data":"b49c76bf77a085fd930b5ec31ac6c10d6d177ab37deb1d881a70d5886058c658"}
Nov 04 00:35:33.906606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.906583    6430 generic.go:356] "Generic (PLEG): container finished" podID="1903fd85-6808-4007-8cec-1aa4a137456c" containerID="6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260" exitCode=0
Nov 04 00:35:33.906749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.906702    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"
Nov 04 00:35:33.906749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.906724    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" event={"ID":"1903fd85-6808-4007-8cec-1aa4a137456c","Type":"ContainerDied","Data":"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"}
Nov 04 00:35:33.906934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.906771    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk" event={"ID":"1903fd85-6808-4007-8cec-1aa4a137456c","Type":"ContainerDied","Data":"0a46d3a16b2d41e7fa6d2b758c8e6ef94f193ec88583a6f945fb09ab6b9fe9bd"}
Nov 04 00:35:33.906934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.906801    6430 scope.go:122] "RemoveContainer" containerID="6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"
Nov 04 00:35:33.910118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.910093    6430 generic.go:356] "Generic (PLEG): container finished" podID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerID="a5810fb9434db5941bc68d70b3143e47694b8651e0967c1ab7d212c398c1027a" exitCode=0
Nov 04 00:35:33.910247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.910147    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" event={"ID":"209517d0-0b9c-4657-8d07-1c74ab5505ed","Type":"ContainerDied","Data":"a5810fb9434db5941bc68d70b3143e47694b8651e0967c1ab7d212c398c1027a"}
Nov 04 00:35:33.912179 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.912155    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 04 00:35:33.925669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.925637    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.925789 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.925716    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.925869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.925819    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.925937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.925875    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.926000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.925985    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.926063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.926052    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.926135 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.926117    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.926197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.926188    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.926253 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.926224    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\") pod \"1903fd85-6808-4007-8cec-1aa4a137456c\" (UID: \"1903fd85-6808-4007-8cec-1aa4a137456c\") "
Nov 04 00:35:33.930109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.927707    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:33.930109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.927810    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:33.930109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.927872    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:33.930109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.927929    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:33.930109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.928239    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:33.930634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.930256    6430 scope.go:122] "RemoveContainer" containerID="6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"
Nov 04 00:35:33.930634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.930455    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:33.931057 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:33.931026    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260\": container with ID starting with 6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260 not found: ID does not exist" containerID="6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"
Nov 04 00:35:33.931172 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.931069    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260"} err="failed to get container status \"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260\": rpc error: code = NotFound desc = could not find container \"6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260\": container with ID starting with 6d2f7bbde32e30fdeb8748198359809511741fbb9ff4e1616bab30f109287260 not found: ID does not exist"
Nov 04 00:35:33.933329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.933301    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:33.933526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.933504    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:33.933950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.933926    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc" pod "1903fd85-6808-4007-8cec-1aa4a137456c" (UID: "1903fd85-6808-4007-8cec-1aa4a137456c"). InnerVolumeSpecName "kube-api-access-gr6lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:33.961739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:33.961716    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:35:34.028984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.028943    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029011    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029070    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029118    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029164 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029156    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029207    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029212    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.029494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029239    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029291    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029494 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029286    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029491    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029617    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029625    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029665    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029708    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.029796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029745    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.030186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029862    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.030186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.030186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029944    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\") pod \"1226101c-28de-4eed-a854-c6db2a7b8540\" (UID: \"1226101c-28de-4eed-a854-c6db2a7b8540\") "
Nov 04 00:35:34.030186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.029988    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\") pod \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\" (UID: \"dcda01ba-ca83-4671-8e3c-b2ffc559f992\") "
Nov 04 00:35:34.030974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.030944    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.030981    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.030997    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031011    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031025    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031038    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1903fd85-6808-4007-8cec-1aa4a137456c-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031051    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031065    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031082    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr6lc\" (UniqueName: \"kubernetes.io/projected/1903fd85-6808-4007-8cec-1aa4a137456c-kube-api-access-gr6lc\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031097    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031113    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1903fd85-6808-4007-8cec-1aa4a137456c-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031310 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031128    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1903fd85-6808-4007-8cec-1aa4a137456c-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.031512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.031450    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032004    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "kube-api-access-pxf56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.032133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032061    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.032206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032123    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:34.032267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032252    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.032453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032430    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032453 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032442    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032657 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032447    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032656    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032672    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.032884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.032860    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:34.033193 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.033167    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:34.033319 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.033297    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "kube-api-access-47xml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.033531 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.033508    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy" pod "dcda01ba-ca83-4671-8e3c-b2ffc559f992" (UID: "dcda01ba-ca83-4671-8e3c-b2ffc559f992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.033835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.033813    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy" pod "1226101c-28de-4eed-a854-c6db2a7b8540" (UID: "1226101c-28de-4eed-a854-c6db2a7b8540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.132371 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132336    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132371 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132367    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132383    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxf56\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-kube-api-access-pxf56\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132417    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132426    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xml\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-kube-api-access-47xml\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132435    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1226101c-28de-4eed-a854-c6db2a7b8540-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132443    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132459    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132474    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132482    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132491    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1226101c-28de-4eed-a854-c6db2a7b8540-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132500    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132509    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1226101c-28de-4eed-a854-c6db2a7b8540-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132518    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda01ba-ca83-4671-8e3c-b2ffc559f992-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.132609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.132526    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1226101c-28de-4eed-a854-c6db2a7b8540-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.163972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.163888    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:35:34.164166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.164118    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" podUID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" containerName="maas-api" containerID="cri-o://73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6" gracePeriod=30
Nov 04 00:35:34.241586 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.241555    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:35:34.249744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.249373    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ht2tk"]
Nov 04 00:35:34.309183 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.309153    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:35:34.414067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.414002    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:35:34.436578 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.436526    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.436924    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437206    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437087    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437258    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437285    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437317    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.437378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437347    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437479    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437583    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437608    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\") pod \"209517d0-0b9c-4657-8d07-1c74ab5505ed\" (UID: \"209517d0-0b9c-4657-8d07-1c74ab5505ed\") "
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437676    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.437794    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.438095    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.438135    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.438949    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.439585    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.440431    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "kube-api-access-zgct5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.441379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.441191    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.442128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.442102    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy" pod "209517d0-0b9c-4657-8d07-1c74ab5505ed" (UID: "209517d0-0b9c-4657-8d07-1c74ab5505ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:34.540429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.540379    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\") pod \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") "
Nov 04 00:35:34.540649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.540557    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\") pod \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") "
Nov 04 00:35:34.540649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.540593    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\" (UniqueName: \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\") pod \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") "
Nov 04 00:35:34.540649 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.540640    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") pod \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\" (UID: \"2c13f4ca-345b-4842-bd79-f341aa0d5efa\") "
Nov 04 00:35:34.540841 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.540770    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca" pod "2c13f4ca-345b-4842-bd79-f341aa0d5efa" (UID: "2c13f4ca-345b-4842-bd79-f341aa0d5efa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:34.541192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541166    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c13f4ca-345b-4842-bd79-f341aa0d5efa-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541192 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541193    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209517d0-0b9c-4657-8d07-1c74ab5505ed-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541206    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541214    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541225    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgct5\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-kube-api-access-zgct5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541234    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209517d0-0b9c-4657-8d07-1c74ab5505ed-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541244    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.541445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.541254    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209517d0-0b9c-4657-8d07-1c74ab5505ed-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.543143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.543112    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls" pod "2c13f4ca-345b-4842-bd79-f341aa0d5efa" (UID: "2c13f4ca-345b-4842-bd79-f341aa0d5efa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:34.543143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.543121    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp" pod "2c13f4ca-345b-4842-bd79-f341aa0d5efa" (UID: "2c13f4ca-345b-4842-bd79-f341aa0d5efa"). InnerVolumeSpecName "kube-api-access-2r2vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:34.543433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.543368    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs" pod "2c13f4ca-345b-4842-bd79-f341aa0d5efa" (UID: "2c13f4ca-345b-4842-bd79-f341aa0d5efa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:34.643169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.643134    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.643169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.643168    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2vp\" (UniqueName: \"kubernetes.io/projected/2c13f4ca-345b-4842-bd79-f341aa0d5efa-kube-api-access-2r2vp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.643346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.643185    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c13f4ca-345b-4842-bd79-f341aa0d5efa-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:34.830322 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:34.830271    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:35:34.918256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.918224    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"
Nov 04 00:35:34.918484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.918223    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8" event={"ID":"1226101c-28de-4eed-a854-c6db2a7b8540","Type":"ContainerDied","Data":"5d841386d98004c5e401df42dea35ec7fed1790541626c721ccd7f8f449777a5"}
Nov 04 00:35:34.918484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.918417    6430 scope.go:122] "RemoveContainer" containerID="b49c76bf77a085fd930b5ec31ac6c10d6d177ab37deb1d881a70d5886058c658"
Nov 04 00:35:34.921136 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.921113    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"
Nov 04 00:35:34.921293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.921151    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2" event={"ID":"209517d0-0b9c-4657-8d07-1c74ab5505ed","Type":"ContainerDied","Data":"a1e878353c2d16834f52a91e8d72bae4c6846e036e104218d2b198db81b499df"}
Nov 04 00:35:34.929744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.929717    6430 generic.go:356] "Generic (PLEG): container finished" podID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" containerID="73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6" exitCode=0
Nov 04 00:35:34.929900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.929772    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"
Nov 04 00:35:34.929900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.929803    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" event={"ID":"2c13f4ca-345b-4842-bd79-f341aa0d5efa","Type":"ContainerDied","Data":"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"}
Nov 04 00:35:34.929900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.929847    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d" event={"ID":"2c13f4ca-345b-4842-bd79-f341aa0d5efa","Type":"ContainerDied","Data":"2f0776f6c75f2e586d7945166eae4752edc2923e3799113a937108c266b78b3a"}
Nov 04 00:35:34.932180 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.931535    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"
Nov 04 00:35:34.932180 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.931571    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c" event={"ID":"dcda01ba-ca83-4671-8e3c-b2ffc559f992","Type":"ContainerDied","Data":"0dc551c1e2b7da7a031f3983df69fc25f2a5d2378418b98e74adca868e7e4987"}
Nov 04 00:35:34.932180 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.932115    6430 scope.go:122] "RemoveContainer" containerID="a5810fb9434db5941bc68d70b3143e47694b8651e0967c1ab7d212c398c1027a"
Nov 04 00:35:34.942253 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.942237    6430 scope.go:122] "RemoveContainer" containerID="73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"
Nov 04 00:35:34.946007 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.945566    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:35:34.952587 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.952551    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q9pm8"]
Nov 04 00:35:34.961302 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.961282    6430 scope.go:122] "RemoveContainer" containerID="73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"
Nov 04 00:35:34.961621 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:34.961596    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6\": container with ID starting with 73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6 not found: ID does not exist" containerID="73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"
Nov 04 00:35:34.961703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.961626    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6"} err="failed to get container status \"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6\": rpc error: code = NotFound desc = could not find container \"73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6\": container with ID starting with 73a4d49da1e06838e8d4c2a87c72e1228e26837c724382cc1de99d426f4867a6 not found: ID does not exist"
Nov 04 00:35:34.961703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.961642    6430 scope.go:122] "RemoveContainer" containerID="f1aca37da1818d8a06da9f5f826890d9a0dbdbe315df15695e2be80ff2ccbfc9"
Nov 04 00:35:34.969415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.969364    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:35:34.973492 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.973472    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-67dv2"]
Nov 04 00:35:34.986979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.986953    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:35:34.991240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:34.991216    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2697a0-75bdf86b9-dsz6d"]
Nov 04 00:35:35.005256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:35.005236    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 04 00:35:35.009001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:35.008980    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-km48c"]
Nov 04 00:35:36.231259 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.231226    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" path="/var/lib/kubelet/pods/1226101c-28de-4eed-a854-c6db2a7b8540/volumes"
Nov 04 00:35:36.231743 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.231727    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" path="/var/lib/kubelet/pods/1903fd85-6808-4007-8cec-1aa4a137456c/volumes"
Nov 04 00:35:36.232147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.232134    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" path="/var/lib/kubelet/pods/209517d0-0b9c-4657-8d07-1c74ab5505ed/volumes"
Nov 04 00:35:36.232811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.232792    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" path="/var/lib/kubelet/pods/2c13f4ca-345b-4842-bd79-f341aa0d5efa/volumes"
Nov 04 00:35:36.233237 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.233224    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" path="/var/lib/kubelet/pods/dcda01ba-ca83-4671-8e3c-b2ffc559f992/volumes"
Nov 04 00:35:36.422054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.422009    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:35:36.422257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:36.422222    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy" containerID="cri-o://1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862" gracePeriod=30
Nov 04 00:35:37.762637 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:37.762600    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:38.771465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:38.771428    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:41.479826 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.479793    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:35:41.680082 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.680057    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:35:41.837321 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837291    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837352    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837382    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837427    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837457    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837495    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837561    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837594    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837639    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\") pod \"8008f1c5-c313-4781-8347-e155daa11c4b\" (UID: \"8008f1c5-c313-4781-8347-e155daa11c4b\") "
Nov 04 00:35:41.837778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837635    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:41.838002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837910    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:41.838002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837944    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:41.838002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837957    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:41.838002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.837997    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:41.838225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.838159    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.838225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.838174    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8008f1c5-c313-4781-8347-e155daa11c4b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.838225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.838182    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.838225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.838191    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.838225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.838208    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.839895 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.839867    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:35:41.839895 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.839879    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:35:41.839895 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.839876    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:41.840089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.839977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56" pod "8008f1c5-c313-4781-8347-e155daa11c4b" (UID: "8008f1c5-c313-4781-8347-e155daa11c4b"). InnerVolumeSpecName "kube-api-access-nsm56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:41.939336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.939297    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8008f1c5-c313-4781-8347-e155daa11c4b-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.939336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.939325    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8008f1c5-c313-4781-8347-e155daa11c4b-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.939336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.939336    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.939581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.939346    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsm56\" (UniqueName: \"kubernetes.io/projected/8008f1c5-c313-4781-8347-e155daa11c4b-kube-api-access-nsm56\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:41.975323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.975284    6430 generic.go:356] "Generic (PLEG): container finished" podID="8008f1c5-c313-4781-8347-e155daa11c4b" containerID="1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862" exitCode=0
Nov 04 00:35:41.975528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.975344    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"
Nov 04 00:35:41.975528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.975369    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerDied","Data":"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"}
Nov 04 00:35:41.975528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.975473    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp" event={"ID":"8008f1c5-c313-4781-8347-e155daa11c4b","Type":"ContainerDied","Data":"9726840fa0dde821fc602cda2b43edf25be43077a0a9b225360e4d922a1fb032"}
Nov 04 00:35:41.975528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.975506    6430 scope.go:122] "RemoveContainer" containerID="1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"
Nov 04 00:35:41.985832 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.985811    6430 scope.go:122] "RemoveContainer" containerID="4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"
Nov 04 00:35:41.994469 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.994449    6430 scope.go:122] "RemoveContainer" containerID="1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"
Nov 04 00:35:41.994727 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:41.994705    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862\": container with ID starting with 1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862 not found: ID does not exist" containerID="1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"
Nov 04 00:35:41.994813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.994739    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862"} err="failed to get container status \"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862\": rpc error: code = NotFound desc = could not find container \"1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862\": container with ID starting with 1633314e6ab9759da4abd129bde8c9746d6c7154290e51faf006057a77911862 not found: ID does not exist"
Nov 04 00:35:41.994813 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.994761    6430 scope.go:122] "RemoveContainer" containerID="4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"
Nov 04 00:35:41.995010 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:41.994991    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a\": container with ID starting with 4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a not found: ID does not exist" containerID="4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"
Nov 04 00:35:41.995051 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:41.995016    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a"} err="failed to get container status \"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a\": rpc error: code = NotFound desc = could not find container \"4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a\": container with ID starting with 4d3a7669a8cdb5300639cb537c82cfeae663db6fcf7b2a1ae74d09ef2bf6847a not found: ID does not exist"
Nov 04 00:35:42.006309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:42.006285    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:35:42.013341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:42.013317    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2697a0-openshift-default-d9b86885c-zfrsp"]
Nov 04 00:35:42.231309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:42.231235    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" path="/var/lib/kubelet/pods/8008f1c5-c313-4781-8347-e155daa11c4b/volumes"
Nov 04 00:35:44.367660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:44.367623    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 04 00:35:44.383154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:44.383112    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 04 00:35:47.762710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:47.762669    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:48.770746 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:48.770711    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:49.368107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:49.368062    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing" containerID="cri-o://64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2" gracePeriod=25
Nov 04 00:35:49.383605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:49.383571    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing" containerID="cri-o://a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec" gracePeriod=25
Nov 04 00:35:50.238566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.238529    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 04 00:35:50.239085 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.238755    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" podUID="9cd8c821-c06d-45e8-a643-4a4825404f30" containerName="maas-api" containerID="cri-o://de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68" gracePeriod=30
Nov 04 00:35:50.523890 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.523863    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 04 00:35:50.647209 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.647171    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\") pod \"9cd8c821-c06d-45e8-a643-4a4825404f30\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") "
Nov 04 00:35:50.647438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.647269    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\") pod \"9cd8c821-c06d-45e8-a643-4a4825404f30\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") "
Nov 04 00:35:50.647438 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.647329    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\") pod \"9cd8c821-c06d-45e8-a643-4a4825404f30\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") "
Nov 04 00:35:50.647600 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.647478    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\" (UniqueName: \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\") pod \"9cd8c821-c06d-45e8-a643-4a4825404f30\" (UID: \"9cd8c821-c06d-45e8-a643-4a4825404f30\") "
Nov 04 00:35:50.648020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.647988    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca" pod "9cd8c821-c06d-45e8-a643-4a4825404f30" (UID: "9cd8c821-c06d-45e8-a643-4a4825404f30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:50.649503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.649471    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls" pod "9cd8c821-c06d-45e8-a643-4a4825404f30" (UID: "9cd8c821-c06d-45e8-a643-4a4825404f30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:50.649699 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.649676    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs" pod "9cd8c821-c06d-45e8-a643-4a4825404f30" (UID: "9cd8c821-c06d-45e8-a643-4a4825404f30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:35:50.651161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.651141    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk" pod "9cd8c821-c06d-45e8-a643-4a4825404f30" (UID: "9cd8c821-c06d-45e8-a643-4a4825404f30"). InnerVolumeSpecName "kube-api-access-5fnmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:50.748976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.748903    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnmk\" (UniqueName: \"kubernetes.io/projected/9cd8c821-c06d-45e8-a643-4a4825404f30-kube-api-access-5fnmk\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:50.748976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.748933    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:50.748976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.748947    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c821-c06d-45e8-a643-4a4825404f30-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:50.748976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:50.748960    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd8c821-c06d-45e8-a643-4a4825404f30-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:51.029120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.029035    6430 generic.go:356] "Generic (PLEG): container finished" podID="9cd8c821-c06d-45e8-a643-4a4825404f30" containerID="de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68" exitCode=0
Nov 04 00:35:51.029120 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.029084    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"
Nov 04 00:35:51.029320 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.029126    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" event={"ID":"9cd8c821-c06d-45e8-a643-4a4825404f30","Type":"ContainerDied","Data":"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"}
Nov 04 00:35:51.029320 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.029208    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v" event={"ID":"9cd8c821-c06d-45e8-a643-4a4825404f30","Type":"ContainerDied","Data":"4db2dff0a3799bcb3b87de434e8bbeff3c6fe9b38c06245b2b2961d72068c42f"}
Nov 04 00:35:51.029320 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.029229    6430 scope.go:122] "RemoveContainer" containerID="de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"
Nov 04 00:35:51.041778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.039826    6430 scope.go:122] "RemoveContainer" containerID="de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"
Nov 04 00:35:51.041778 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:35:51.040236    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68\": container with ID starting with de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68 not found: ID does not exist" containerID="de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"
Nov 04 00:35:51.041778 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.040267    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68"} err="failed to get container status \"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68\": rpc error: code = NotFound desc = could not find container \"de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68\": container with ID starting with de6e9462de43ab3e3446954f924faeaa53da19d520dbb50d0e1adc198402bf68 not found: ID does not exist"
Nov 04 00:35:51.054601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.054573    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 04 00:35:51.061683 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:51.061659    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4331a7-8648c9fd7f-jbd2v"]
Nov 04 00:35:52.230790 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:52.230756    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd8c821-c06d-45e8-a643-4a4825404f30" path="/var/lib/kubelet/pods/9cd8c821-c06d-45e8-a643-4a4825404f30/volumes"
Nov 04 00:35:54.050052 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.050024    6430 generic.go:356] "Generic (PLEG): container finished" podID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerID="e2f546c75723787bd12243f5d09a173030eef39e9ba7261f0e8b878767a3d213" exitCode=0
Nov 04 00:35:54.050486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.050101    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" event={"ID":"41c967b0-d1e9-4e4e-a4af-b613ab839c7d","Type":"ContainerDied","Data":"e2f546c75723787bd12243f5d09a173030eef39e9ba7261f0e8b878767a3d213"}
Nov 04 00:35:54.051943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.051917    6430 generic.go:356] "Generic (PLEG): container finished" podID="02fea092-5031-4ce0-b79f-42501c49a619" containerID="02b81c304946e46f828e98c878698b79c9100e7aaeb2a8c1c57d06ca200fb5b7" exitCode=0
Nov 04 00:35:54.052074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.051989    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" event={"ID":"02fea092-5031-4ce0-b79f-42501c49a619","Type":"ContainerDied","Data":"02b81c304946e46f828e98c878698b79c9100e7aaeb2a8c1c57d06ca200fb5b7"}
Nov 04 00:35:54.080041 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.080020    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 04 00:35:54.083489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.083470    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 04 00:35:54.181937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.181903    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\" (UniqueName: \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\") pod \"02fea092-5031-4ce0-b79f-42501c49a619\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") "
Nov 04 00:35:54.182115 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182047    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") pod \"02fea092-5031-4ce0-b79f-42501c49a619\" (UID: \"02fea092-5031-4ce0-b79f-42501c49a619\") "
Nov 04 00:35:54.182218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182202    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\") pod \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") "
Nov 04 00:35:54.182288 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182235    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\" (UniqueName: \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\") pod \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\" (UID: \"41c967b0-d1e9-4e4e-a4af-b613ab839c7d\") "
Nov 04 00:35:54.182563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182435    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume" pod "02fea092-5031-4ce0-b79f-42501c49a619" (UID: "02fea092-5031-4ce0-b79f-42501c49a619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:54.182650 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182603    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume" pod "41c967b0-d1e9-4e4e-a4af-b613ab839c7d" (UID: "41c967b0-d1e9-4e4e-a4af-b613ab839c7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:35:54.182702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.182659    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fea092-5031-4ce0-b79f-42501c49a619-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:54.184190 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.184168    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6" pod "02fea092-5031-4ce0-b79f-42501c49a619" (UID: "02fea092-5031-4ce0-b79f-42501c49a619"). InnerVolumeSpecName "kube-api-access-mbjz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:54.184625 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.184605    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl" pod "41c967b0-d1e9-4e4e-a4af-b613ab839c7d" (UID: "41c967b0-d1e9-4e4e-a4af-b613ab839c7d"). InnerVolumeSpecName "kube-api-access-mh6rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:35:54.283791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.283757    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:54.283791 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.283784    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6rl\" (UniqueName: \"kubernetes.io/projected/41c967b0-d1e9-4e4e-a4af-b613ab839c7d-kube-api-access-mh6rl\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:54.284029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.283798    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjz6\" (UniqueName: \"kubernetes.io/projected/02fea092-5031-4ce0-b79f-42501c49a619-kube-api-access-mbjz6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:35:54.917175 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.917122    6430 scope.go:122] "RemoveContainer" containerID="e2f546c75723787bd12243f5d09a173030eef39e9ba7261f0e8b878767a3d213"
Nov 04 00:35:54.926265 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:54.926242    6430 scope.go:122] "RemoveContainer" containerID="02b81c304946e46f828e98c878698b79c9100e7aaeb2a8c1c57d06ca200fb5b7"
Nov 04 00:35:55.058069 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.058018    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx" event={"ID":"41c967b0-d1e9-4e4e-a4af-b613ab839c7d","Type":"ContainerDied","Data":"0d66795a0179bcc8c52568e04365bd26c7e3178e42844c465a263d14922c2f5b"}
Nov 04 00:35:55.058539 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.058086    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8" event={"ID":"02fea092-5031-4ce0-b79f-42501c49a619","Type":"ContainerDied","Data":"3fe73948a1d4725a988196ddc4e154da734f93673a5984061c44d343476fdaf5"}
Nov 04 00:35:55.058539 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.058092    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"
Nov 04 00:35:55.058539 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.058092    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"
Nov 04 00:35:55.117596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.117563    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 04 00:35:55.122199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.122171    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-58b75d-757b846954-766kx"]
Nov 04 00:35:55.139902 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.139879    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 04 00:35:55.147526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:55.147494    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-58b75d-bb6b7999f-lccl8"]
Nov 04 00:35:56.232817 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:56.232779    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fea092-5031-4ce0-b79f-42501c49a619" path="/var/lib/kubelet/pods/02fea092-5031-4ce0-b79f-42501c49a619/volumes"
Nov 04 00:35:56.233241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:56.233226    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" path="/var/lib/kubelet/pods/41c967b0-d1e9-4e4e-a4af-b613ab839c7d/volumes"
Nov 04 00:35:57.752170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:57.752130    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:57.752614 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:57.752135    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:35:57.851703 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:35:57.851666    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:36:02.567341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:02.567305    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 04 00:36:02.568004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:02.567613    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy" containerID="cri-o://e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c" gracePeriod=30
Nov 04 00:36:04.830862 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:04.830822    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:36:07.752247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.752209    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:07.752729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.752684    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:07.836216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.836186    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 04 00:36:07.949581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949504    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949539    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949557    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949621    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949649    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949674    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.949848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949773    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949816    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949875    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\") pod \"46731542-c799-4992-89c3-221d1bd1c18a\" (UID: \"46731542-c799-4992-89c3-221d1bd1c18a\") "
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949875    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.949981    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:07.950101 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950026    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:07.950432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950138    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:07.950499 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950467    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:07.950563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950506    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:07.950563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950545    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:07.950669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950562    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46731542-c799-4992-89c3-221d1bd1c18a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:07.950669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.950576    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:07.952061 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.952037    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:36:07.952202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.952099    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:07.952293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.952271    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:07.952359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:07.952297    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn" pod "46731542-c799-4992-89c3-221d1bd1c18a" (UID: "46731542-c799-4992-89c3-221d1bd1c18a"). InnerVolumeSpecName "kube-api-access-r4jkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:08.052029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.051997    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.052029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.052025    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jkn\" (UniqueName: \"kubernetes.io/projected/46731542-c799-4992-89c3-221d1bd1c18a-kube-api-access-r4jkn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.052029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.052036    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46731542-c799-4992-89c3-221d1bd1c18a-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.052284 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.052045    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46731542-c799-4992-89c3-221d1bd1c18a-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:08.130551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.130513    6430 generic.go:356] "Generic (PLEG): container finished" podID="46731542-c799-4992-89c3-221d1bd1c18a" containerID="e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c" exitCode=0
Nov 04 00:36:08.130716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.130573    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"
Nov 04 00:36:08.130716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.130587    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerDied","Data":"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"}
Nov 04 00:36:08.130716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.130644    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58" event={"ID":"46731542-c799-4992-89c3-221d1bd1c18a","Type":"ContainerDied","Data":"3ec5559bf1c7c3a8020f0a6f13b9aec100287edcf2787c44bb01ec2a5c7110fa"}
Nov 04 00:36:08.130716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.130667    6430 scope.go:122] "RemoveContainer" containerID="e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"
Nov 04 00:36:08.140617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.140600    6430 scope.go:122] "RemoveContainer" containerID="9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"
Nov 04 00:36:08.149003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.148986    6430 scope.go:122] "RemoveContainer" containerID="e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"
Nov 04 00:36:08.149248 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:08.149230    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c\": container with ID starting with e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c not found: ID does not exist" containerID="e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"
Nov 04 00:36:08.149330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.149257    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c"} err="failed to get container status \"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c\": rpc error: code = NotFound desc = could not find container \"e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c\": container with ID starting with e0cf5d50bf403806d1e50d20174747b41722f7758f64539b6fdc4478190c877c not found: ID does not exist"
Nov 04 00:36:08.149330 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.149274    6430 scope.go:122] "RemoveContainer" containerID="9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"
Nov 04 00:36:08.149550 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:08.149528    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576\": container with ID starting with 9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576 not found: ID does not exist" containerID="9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"
Nov 04 00:36:08.149611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.149559    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576"} err="failed to get container status \"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576\": rpc error: code = NotFound desc = could not find container \"9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576\": container with ID starting with 9fc82d36b6ee3f9b7715465721a23c985c7089acc278825ddd4421ce92f05576 not found: ID does not exist"
Nov 04 00:36:08.155171 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.155148    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 04 00:36:08.160902 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.160879    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4331a7-openshift-default-7658bdf747-9rh58"]
Nov 04 00:36:08.231692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.231621    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46731542-c799-4992-89c3-221d1bd1c18a" path="/var/lib/kubelet/pods/46731542-c799-4992-89c3-221d1bd1c18a/volumes"
Nov 04 00:36:08.703972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.703933    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 04 00:36:08.721309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:08.721276    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 04 00:36:09.390852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.390820    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 04 00:36:09.651950 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.651923    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 04 00:36:09.687836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.687811    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 04 00:36:09.779819 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.779786    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\") pod \"004957b7-0d92-42cc-a846-876ac4dfd3cf\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") "
Nov 04 00:36:09.779819 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.779820    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\" (UniqueName: \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\") pod \"004957b7-0d92-42cc-a846-876ac4dfd3cf\" (UID: \"004957b7-0d92-42cc-a846-876ac4dfd3cf\") "
Nov 04 00:36:09.780037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.779849    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\" (UniqueName: \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\") pod \"499d212b-e9d7-424e-92cf-07b3f681208f\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") "
Nov 04 00:36:09.780037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.779954    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\") pod \"499d212b-e9d7-424e-92cf-07b3f681208f\" (UID: \"499d212b-e9d7-424e-92cf-07b3f681208f\") "
Nov 04 00:36:09.780351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.780323    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume" pod "004957b7-0d92-42cc-a846-876ac4dfd3cf" (UID: "004957b7-0d92-42cc-a846-876ac4dfd3cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:09.780462 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.780359    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume" pod "499d212b-e9d7-424e-92cf-07b3f681208f" (UID: "499d212b-e9d7-424e-92cf-07b3f681208f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:09.780462 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.780456    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004957b7-0d92-42cc-a846-876ac4dfd3cf-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.780592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.780470    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/499d212b-e9d7-424e-92cf-07b3f681208f-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.782001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.781977    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx" pod "499d212b-e9d7-424e-92cf-07b3f681208f" (UID: "499d212b-e9d7-424e-92cf-07b3f681208f"). InnerVolumeSpecName "kube-api-access-8jcgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:09.782113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.782002    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6" pod "004957b7-0d92-42cc-a846-876ac4dfd3cf" (UID: "004957b7-0d92-42cc-a846-876ac4dfd3cf"). InnerVolumeSpecName "kube-api-access-rsrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:09.881360 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.881281    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrf6\" (UniqueName: \"kubernetes.io/projected/004957b7-0d92-42cc-a846-876ac4dfd3cf-kube-api-access-rsrf6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:09.881360 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:09.881308    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcgx\" (UniqueName: \"kubernetes.io/projected/499d212b-e9d7-424e-92cf-07b3f681208f-kube-api-access-8jcgx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:10.144627 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.144547    6430 generic.go:356] "Generic (PLEG): container finished" podID="499d212b-e9d7-424e-92cf-07b3f681208f" containerID="a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec" exitCode=0
Nov 04 00:36:10.144627 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.144593    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"
Nov 04 00:36:10.144847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.144626    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" event={"ID":"499d212b-e9d7-424e-92cf-07b3f681208f","Type":"ContainerDied","Data":"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"}
Nov 04 00:36:10.144847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.144678    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl" event={"ID":"499d212b-e9d7-424e-92cf-07b3f681208f","Type":"ContainerDied","Data":"da15a918f0dd0b35dffa2af3e896c976fbafbd88af58c88c62984f39c31a2eb6"}
Nov 04 00:36:10.144847 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.144699    6430 scope.go:122] "RemoveContainer" containerID="a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"
Nov 04 00:36:10.145998 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.145975    6430 generic.go:356] "Generic (PLEG): container finished" podID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerID="64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2" exitCode=0
Nov 04 00:36:10.146128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.146029    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"
Nov 04 00:36:10.146128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.146065    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" event={"ID":"004957b7-0d92-42cc-a846-876ac4dfd3cf","Type":"ContainerDied","Data":"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"}
Nov 04 00:36:10.146128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.146104    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt" event={"ID":"004957b7-0d92-42cc-a846-876ac4dfd3cf","Type":"ContainerDied","Data":"94682d717d9e821989eaf6af6597f609c7f329d65f20c5eb952ef7db6ce37ef0"}
Nov 04 00:36:10.154278 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.154262    6430 scope.go:122] "RemoveContainer" containerID="a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"
Nov 04 00:36:10.154591 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:10.154569    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec\": container with ID starting with a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec not found: ID does not exist" containerID="a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"
Nov 04 00:36:10.154696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.154596    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec"} err="failed to get container status \"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec\": rpc error: code = NotFound desc = could not find container \"a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec\": container with ID starting with a6c7f41ce470b1c5e71ff47db59f5c31a1e568826b28f6fe13cecfbfc192fcec not found: ID does not exist"
Nov 04 00:36:10.154696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.154613    6430 scope.go:122] "RemoveContainer" containerID="64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"
Nov 04 00:36:10.163087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.163064    6430 scope.go:122] "RemoveContainer" containerID="64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"
Nov 04 00:36:10.163324 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:10.163304    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2\": container with ID starting with 64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2 not found: ID does not exist" containerID="64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"
Nov 04 00:36:10.163408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.163335    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2"} err="failed to get container status \"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2\": rpc error: code = NotFound desc = could not find container \"64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2\": container with ID starting with 64087afdf3e65a84eeecc63cf3e26a0f4a9907be118b760f1551f78856c7b2f2 not found: ID does not exist"
Nov 04 00:36:10.173680 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.173623    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 04 00:36:10.175843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.175818    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4331a7-7c66c6c8cb-8rvlt"]
Nov 04 00:36:10.188702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.188682    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 04 00:36:10.192414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.192379    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4331a7-7c79d79747-ln4vl"]
Nov 04 00:36:10.230267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.230238    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" path="/var/lib/kubelet/pods/004957b7-0d92-42cc-a846-876ac4dfd3cf/volumes"
Nov 04 00:36:10.230755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:10.230733    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" path="/var/lib/kubelet/pods/499d212b-e9d7-424e-92cf-07b3f681208f/volumes"
Nov 04 00:36:12.518712 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:12.518677    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:36:13.705230 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:13.705187    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing" containerID="cri-o://13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309" gracePeriod=25
Nov 04 00:36:13.721757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:13.721717    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing" containerID="cri-o://1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe" gracePeriod=25
Nov 04 00:36:13.829212 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:13.829158    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 04 00:36:13.829478 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:13.829445    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" podUID="ff8bc764-815f-4618-8d83-cba9e0d57def" containerName="maas-api" containerID="cri-o://7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad" gracePeriod=30
Nov 04 00:36:14.092287 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.092260    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 04 00:36:14.131154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.131119    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\") pod \"ff8bc764-815f-4618-8d83-cba9e0d57def\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") "
Nov 04 00:36:14.131154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.131158    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") pod \"ff8bc764-815f-4618-8d83-cba9e0d57def\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") "
Nov 04 00:36:14.131400 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.131193    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\") pod \"ff8bc764-815f-4618-8d83-cba9e0d57def\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") "
Nov 04 00:36:14.131589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.131567    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca" pod "ff8bc764-815f-4618-8d83-cba9e0d57def" (UID: "ff8bc764-815f-4618-8d83-cba9e0d57def"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:14.131729 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.131710    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8bc764-815f-4618-8d83-cba9e0d57def-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:14.133319 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.133292    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs" pod "ff8bc764-815f-4618-8d83-cba9e0d57def" (UID: "ff8bc764-815f-4618-8d83-cba9e0d57def"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:14.133426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.133316    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls" pod "ff8bc764-815f-4618-8d83-cba9e0d57def" (UID: "ff8bc764-815f-4618-8d83-cba9e0d57def"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:14.172859 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.172827    6430 generic.go:356] "Generic (PLEG): container finished" podID="ff8bc764-815f-4618-8d83-cba9e0d57def" containerID="7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad" exitCode=0
Nov 04 00:36:14.172975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.172878    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"
Nov 04 00:36:14.172975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.172883    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" event={"ID":"ff8bc764-815f-4618-8d83-cba9e0d57def","Type":"ContainerDied","Data":"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"}
Nov 04 00:36:14.172975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.172937    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf" event={"ID":"ff8bc764-815f-4618-8d83-cba9e0d57def","Type":"ContainerDied","Data":"af04338577a7065cec3e9dfd7b2b24053e75a436e1038d5309b775610048594b"}
Nov 04 00:36:14.172975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.172968    6430 scope.go:122] "RemoveContainer" containerID="7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"
Nov 04 00:36:14.183341 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.183323    6430 scope.go:122] "RemoveContainer" containerID="7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"
Nov 04 00:36:14.183616 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:14.183598    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad\": container with ID starting with 7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad not found: ID does not exist" containerID="7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"
Nov 04 00:36:14.183674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.183626    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad"} err="failed to get container status \"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad\": rpc error: code = NotFound desc = could not find container \"7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad\": container with ID starting with 7accdc6afee2464ec69118127861e0e58d69b5352b1729349772b283c3832aad not found: ID does not exist"
Nov 04 00:36:14.233652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.232894    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\") pod \"ff8bc764-815f-4618-8d83-cba9e0d57def\" (UID: \"ff8bc764-815f-4618-8d83-cba9e0d57def\") "
Nov 04 00:36:14.233652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.233443    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:14.233652 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.233467    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff8bc764-815f-4618-8d83-cba9e0d57def-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:14.235916 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.235881    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7" pod "ff8bc764-815f-4618-8d83-cba9e0d57def" (UID: "ff8bc764-815f-4618-8d83-cba9e0d57def"). InnerVolumeSpecName "kube-api-access-whrg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:14.334646 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.334615    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrg7\" (UniqueName: \"kubernetes.io/projected/ff8bc764-815f-4618-8d83-cba9e0d57def-kube-api-access-whrg7\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:14.498397 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.498353    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 04 00:36:14.502643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:14.502616    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-398158-dbf6fdd9c-8cskf"]
Nov 04 00:36:16.232437 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:16.232404    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8bc764-815f-4618-8d83-cba9e0d57def" path="/var/lib/kubelet/pods/ff8bc764-815f-4618-8d83-cba9e0d57def/volumes"
Nov 04 00:36:18.519973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.519931    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:36:18.520618 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.520193    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" podUID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" containerName="maas-api" containerID="cri-o://a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3" gracePeriod=30
Nov 04 00:36:18.771596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.771352    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:18.771596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.771381    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:18.801958 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.801930    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:36:18.895471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.895382    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\") pod \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") "
Nov 04 00:36:18.895688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.895497    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") pod \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") "
Nov 04 00:36:18.895688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.895537    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\" (UniqueName: \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\") pod \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") "
Nov 04 00:36:18.895836 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.895687    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\") pod \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\" (UID: \"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe\") "
Nov 04 00:36:18.896100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.896072    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca" pod "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" (UID: "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:18.897716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.897677    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs" pod "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" (UID: "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:18.897716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.897700    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls" pod "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" (UID: "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:18.897853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.897727    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p" pod "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" (UID: "52fdf076-ffef-4a57-9630-4f6e0f5a8ebe"). InnerVolumeSpecName "kube-api-access-gvc4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:18.997137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.997099    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:18.997137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.997133    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:18.997137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.997142    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:18.997424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:18.997154    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvc4p\" (UniqueName: \"kubernetes.io/projected/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe-kube-api-access-gvc4p\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:19.202080 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.202038    6430 generic.go:356] "Generic (PLEG): container finished" podID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" containerID="a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3" exitCode=0
Nov 04 00:36:19.202295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.202087    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"
Nov 04 00:36:19.202295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.202124    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" event={"ID":"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe","Type":"ContainerDied","Data":"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"}
Nov 04 00:36:19.202295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.202184    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc" event={"ID":"52fdf076-ffef-4a57-9630-4f6e0f5a8ebe","Type":"ContainerDied","Data":"2d90d20ef628c7ff1bc79a52980c3023fe86bc2456a679ab27205d4a25c264f9"}
Nov 04 00:36:19.202295 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.202210    6430 scope.go:122] "RemoveContainer" containerID="a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"
Nov 04 00:36:19.213795 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.213768    6430 scope.go:122] "RemoveContainer" containerID="a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"
Nov 04 00:36:19.214133 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:19.214110    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3\": container with ID starting with a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3 not found: ID does not exist" containerID="a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"
Nov 04 00:36:19.214228 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.214142    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3"} err="failed to get container status \"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3\": rpc error: code = NotFound desc = could not find container \"a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3\": container with ID starting with a87323feb2891012837ab581942680659d16f385707f45452b4edc15987d1ef3 not found: ID does not exist"
Nov 04 00:36:19.228547 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.228520    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:36:19.233031 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:19.233005    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8895cb-6d6548c68d-bglzc"]
Nov 04 00:36:20.230733 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:20.230688    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" path="/var/lib/kubelet/pods/52fdf076-ffef-4a57-9630-4f6e0f5a8ebe/volumes"
Nov 04 00:36:21.268104 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:21.268070    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 04 00:36:21.268553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:21.268302    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy" containerID="cri-o://02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5" gracePeriod=30
Nov 04 00:36:24.412309 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.412258    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" containerID="cri-o://a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6" gracePeriod=45
Nov 04 00:36:24.500959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.500917    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Nov 04 00:36:24.671247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.671188    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 04 00:36:24.764504 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764471    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.764714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764567    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.764714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764639    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\" (UniqueName: \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.764849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764717    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.764849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.764849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764770    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.765026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.764860    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\") pod \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\" (UID: \"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed\") "
Nov 04 00:36:24.765853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.765171    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.765853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.765451    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.765853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.765486    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.765853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.765509    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.767664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.767628    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:24.767955 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.767921    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "kube-api-access-tcjkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:24.769343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.769309    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs" pod "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" (UID: "8d7b1139-91b7-44e9-a78f-e8f4d46e83ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:24.867081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.867044    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.867081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.867077    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.867081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.867087    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:24.867323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:24.867097    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcjkn\" (UniqueName: \"kubernetes.io/projected/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed-kube-api-access-tcjkn\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:25.243827 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.243788    6430 generic.go:356] "Generic (PLEG): container finished" podID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerID="a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6" exitCode=0
Nov 04 00:36:25.244015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.243836    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"
Nov 04 00:36:25.244015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.243833    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" event={"ID":"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed","Type":"ContainerDied","Data":"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"}
Nov 04 00:36:25.244015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.243949    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z" event={"ID":"8d7b1139-91b7-44e9-a78f-e8f4d46e83ed","Type":"ContainerDied","Data":"7e5bcc1e1a5e20b033727e166f560da171212e84e026b5903d15f7e8b4515bae"}
Nov 04 00:36:25.244015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.243973    6430 scope.go:122] "RemoveContainer" containerID="a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"
Nov 04 00:36:25.256986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.256967    6430 scope.go:122] "RemoveContainer" containerID="a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"
Nov 04 00:36:25.257280 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:25.257254    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6\": container with ID starting with a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6 not found: ID does not exist" containerID="a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"
Nov 04 00:36:25.257382 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.257290    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6"} err="failed to get container status \"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6\": rpc error: code = NotFound desc = could not find container \"a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6\": container with ID starting with a1c197c0227bc4937bce9bc1b74ed1bee11baa6fce1ea52c554cecbe91719bd6 not found: ID does not exist"
Nov 04 00:36:25.288566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.288537    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 04 00:36:25.298496 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:25.298470    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-2-w2-5738a9-kserve-c75fdbb77-4dm5z"]
Nov 04 00:36:26.233520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.233481    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" path="/var/lib/kubelet/pods/8d7b1139-91b7-44e9-a78f-e8f4d46e83ed/volumes"
Nov 04 00:36:26.541019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.540992    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 04 00:36:26.687715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687672    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.687715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687716    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.687942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687762    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.687942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687825    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.687942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687910    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.687942 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687926    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.688150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687955    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.688150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.687990    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.688150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688019    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\") pod \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\" (UID: \"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd\") "
Nov 04 00:36:26.688150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688055    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:26.688150 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688124    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:26.688452 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688291    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:26.688550 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688527    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.688619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688554    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.688619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688572    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.688619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688605    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:26.688783 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.688657    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:26.690412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.690351    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:26.690546 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.690456    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:36:26.690737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.690716    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "kube-api-access-6nv2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:26.690816 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.690802    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token" pod "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" (UID: "f5fee7e4-0040-404c-8305-fd7cbbb1c5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789518    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789552    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789563    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789571    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789581    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nv2j\" (UniqueName: \"kubernetes.io/projected/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-kube-api-access-6nv2j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:26.789604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:26.789591    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:27.257547 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.257510    6430 generic.go:356] "Generic (PLEG): container finished" podID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerID="02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5" exitCode=0
Nov 04 00:36:27.257987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.257563    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"
Nov 04 00:36:27.257987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.257589    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerDied","Data":"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"}
Nov 04 00:36:27.257987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.257644    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm" event={"ID":"f5fee7e4-0040-404c-8305-fd7cbbb1c5cd","Type":"ContainerDied","Data":"8e97e5425bb40079848192d6083939eb37e2a572340c8a939717d486dab95516"}
Nov 04 00:36:27.257987 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.257673    6430 scope.go:122] "RemoveContainer" containerID="02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"
Nov 04 00:36:27.267897 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.267877    6430 scope.go:122] "RemoveContainer" containerID="f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"
Nov 04 00:36:27.277088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.277069    6430 scope.go:122] "RemoveContainer" containerID="02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"
Nov 04 00:36:27.277369 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:27.277349    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5\": container with ID starting with 02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5 not found: ID does not exist" containerID="02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"
Nov 04 00:36:27.277450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.277379    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5"} err="failed to get container status \"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5\": rpc error: code = NotFound desc = could not find container \"02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5\": container with ID starting with 02c2c710920d8701ebe132c2aa3e653a276beaf284e017aa8f640a930e3815f5 not found: ID does not exist"
Nov 04 00:36:27.277450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.277422    6430 scope.go:122] "RemoveContainer" containerID="f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"
Nov 04 00:36:27.277675 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:27.277657    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475\": container with ID starting with f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475 not found: ID does not exist" containerID="f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"
Nov 04 00:36:27.277714 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.277681    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475"} err="failed to get container status \"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475\": rpc error: code = NotFound desc = could not find container \"f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475\": container with ID starting with f884ffaae8be584f1d1a2574ae866650462db2738fedddf5637699676dc03475 not found: ID does not exist"
Nov 04 00:36:27.283934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.283904    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 04 00:36:27.288201 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:27.288176    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-398158-openshift-default-64cc9466bc-xz4sm"]
Nov 04 00:36:28.231083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:28.231050    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" path="/var/lib/kubelet/pods/f5fee7e4-0040-404c-8305-fd7cbbb1c5cd/volumes"
Nov 04 00:36:28.770698 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:28.770650    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:28.771110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:28.770670    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:36:33.972706 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:33.972639    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 04 00:36:34.003952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.003926    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 04 00:36:34.071060 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.071029    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\") pod \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") "
Nov 04 00:36:34.071247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.071085    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\" (UniqueName: \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\") pod \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\" (UID: \"f2b8ff2b-c457-417c-a7ed-76fba58bd015\") "
Nov 04 00:36:34.071548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.071519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume" pod "f2b8ff2b-c457-417c-a7ed-76fba58bd015" (UID: "f2b8ff2b-c457-417c-a7ed-76fba58bd015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:34.073291 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.073270    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj" pod "f2b8ff2b-c457-417c-a7ed-76fba58bd015" (UID: "f2b8ff2b-c457-417c-a7ed-76fba58bd015"). InnerVolumeSpecName "kube-api-access-7t9lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:34.172533 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.172499    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\" (UniqueName: \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\") pod \"605c8350-f8cd-4208-8185-bfcbaa38856f\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") "
Nov 04 00:36:34.172533 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.172539    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\") pod \"605c8350-f8cd-4208-8185-bfcbaa38856f\" (UID: \"605c8350-f8cd-4208-8185-bfcbaa38856f\") "
Nov 04 00:36:34.172921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.172842    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8ff2b-c457-417c-a7ed-76fba58bd015-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:34.172921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.172855    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9lj\" (UniqueName: \"kubernetes.io/projected/f2b8ff2b-c457-417c-a7ed-76fba58bd015-kube-api-access-7t9lj\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:34.173046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.173005    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume" pod "605c8350-f8cd-4208-8185-bfcbaa38856f" (UID: "605c8350-f8cd-4208-8185-bfcbaa38856f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:34.174561 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.174541    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp" pod "605c8350-f8cd-4208-8185-bfcbaa38856f" (UID: "605c8350-f8cd-4208-8185-bfcbaa38856f"). InnerVolumeSpecName "kube-api-access-mjclp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:34.274275 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.274246    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjclp\" (UniqueName: \"kubernetes.io/projected/605c8350-f8cd-4208-8185-bfcbaa38856f-kube-api-access-mjclp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:34.274275 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.274274    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605c8350-f8cd-4208-8185-bfcbaa38856f-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:34.297624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.297592    6430 generic.go:356] "Generic (PLEG): container finished" podID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerID="13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309" exitCode=0
Nov 04 00:36:34.297773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.297652    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"
Nov 04 00:36:34.297773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.297670    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" event={"ID":"f2b8ff2b-c457-417c-a7ed-76fba58bd015","Type":"ContainerDied","Data":"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"}
Nov 04 00:36:34.297773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.297724    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn" event={"ID":"f2b8ff2b-c457-417c-a7ed-76fba58bd015","Type":"ContainerDied","Data":"92dc1b45728fa90af6ec326d6d0ba49c4ae051af54edd6043b64290ff25014a3"}
Nov 04 00:36:34.297773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.297746    6430 scope.go:122] "RemoveContainer" containerID="13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"
Nov 04 00:36:34.299220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.299187    6430 generic.go:356] "Generic (PLEG): container finished" podID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerID="1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe" exitCode=0
Nov 04 00:36:34.299220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.299226    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" event={"ID":"605c8350-f8cd-4208-8185-bfcbaa38856f","Type":"ContainerDied","Data":"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"}
Nov 04 00:36:34.299368 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.299245    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"
Nov 04 00:36:34.299368 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.299263    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw" event={"ID":"605c8350-f8cd-4208-8185-bfcbaa38856f","Type":"ContainerDied","Data":"33b93abc5a720c165ae8e74fb96f25460e0cc15d5b8511fb251fac1a5fd9f099"}
Nov 04 00:36:34.306734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.306712    6430 scope.go:122] "RemoveContainer" containerID="13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"
Nov 04 00:36:34.307001 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:34.306982    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309\": container with ID starting with 13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309 not found: ID does not exist" containerID="13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"
Nov 04 00:36:34.307075 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.307008    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309"} err="failed to get container status \"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309\": rpc error: code = NotFound desc = could not find container \"13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309\": container with ID starting with 13e8b68b8ca7c4f59741ff73d63d2bc0feef15354e3b6840a22e293f8cbfd309 not found: ID does not exist"
Nov 04 00:36:34.307075 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.307023    6430 scope.go:122] "RemoveContainer" containerID="1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"
Nov 04 00:36:34.315372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.315351    6430 scope.go:122] "RemoveContainer" containerID="1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"
Nov 04 00:36:34.315659 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:34.315641    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe\": container with ID starting with 1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe not found: ID does not exist" containerID="1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"
Nov 04 00:36:34.315715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.315665    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe"} err="failed to get container status \"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe\": rpc error: code = NotFound desc = could not find container \"1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe\": container with ID starting with 1c2b6d71b4267e317583270a3ef3c5d8d60a5d7a9e21c487a8be92d56ecd6bbe not found: ID does not exist"
Nov 04 00:36:34.321196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.321173    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 04 00:36:34.323855 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.323834    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-398158-9666464b4-xnrwn"]
Nov 04 00:36:34.337347 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.337326    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 04 00:36:34.341959 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:34.341937    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-398158-7c74489687-j25bw"]
Nov 04 00:36:34.830273 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:34.830236    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:36:36.231731 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.231693    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" path="/var/lib/kubelet/pods/605c8350-f8cd-4208-8185-bfcbaa38856f/volumes"
Nov 04 00:36:36.232305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.232282    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" path="/var/lib/kubelet/pods/f2b8ff2b-c457-417c-a7ed-76fba58bd015/volumes"
Nov 04 00:36:36.508007 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.507912    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:36:36.508195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.508158    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" podUID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" containerName="maas-api" containerID="cri-o://68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec" gracePeriod=30
Nov 04 00:36:36.724446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.721709    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 04 00:36:36.784639 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.784608    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:36:36.799660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.799623    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") pod \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") "
Nov 04 00:36:36.799838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.799723    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\") pod \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") "
Nov 04 00:36:36.799838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.799778    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\" (UniqueName: \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\") pod \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") "
Nov 04 00:36:36.799838 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.799823    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\") pod \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\" (UID: \"871e77b8-0d34-4fdc-aeac-1b110a9a1eba\") "
Nov 04 00:36:36.800167 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.800138    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca" pod "871e77b8-0d34-4fdc-aeac-1b110a9a1eba" (UID: "871e77b8-0d34-4fdc-aeac-1b110a9a1eba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:36.800315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.800293    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:36.802441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.802413    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g" pod "871e77b8-0d34-4fdc-aeac-1b110a9a1eba" (UID: "871e77b8-0d34-4fdc-aeac-1b110a9a1eba"). InnerVolumeSpecName "kube-api-access-g4h5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:36.802567 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.802465    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls" pod "871e77b8-0d34-4fdc-aeac-1b110a9a1eba" (UID: "871e77b8-0d34-4fdc-aeac-1b110a9a1eba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:36.802852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.802829    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs" pod "871e77b8-0d34-4fdc-aeac-1b110a9a1eba" (UID: "871e77b8-0d34-4fdc-aeac-1b110a9a1eba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:36.901745 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.901696    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:36.901960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.901757    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4h5g\" (UniqueName: \"kubernetes.io/projected/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-kube-api-access-g4h5g\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:36.901960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:36.901774    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871e77b8-0d34-4fdc-aeac-1b110a9a1eba-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:37.225731 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.225691    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:36:37.317460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.317422    6430 generic.go:356] "Generic (PLEG): container finished" podID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" containerID="68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec" exitCode=0
Nov 04 00:36:37.317460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.317455    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" event={"ID":"871e77b8-0d34-4fdc-aeac-1b110a9a1eba","Type":"ContainerDied","Data":"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"}
Nov 04 00:36:37.317996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.317482    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"
Nov 04 00:36:37.317996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.317515    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq" event={"ID":"871e77b8-0d34-4fdc-aeac-1b110a9a1eba","Type":"ContainerDied","Data":"322f85f2aef45ccaff3f324aebd3f4b300f87c6aafda01a4476ddd7ab61de90a"}
Nov 04 00:36:37.317996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.317537    6430 scope.go:122] "RemoveContainer" containerID="68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"
Nov 04 00:36:37.327770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.327752    6430 scope.go:122] "RemoveContainer" containerID="68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"
Nov 04 00:36:37.328039 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:37.328020    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec\": container with ID starting with 68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec not found: ID does not exist" containerID="68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"
Nov 04 00:36:37.328089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.328048    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec"} err="failed to get container status \"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec\": rpc error: code = NotFound desc = could not find container \"68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec\": container with ID starting with 68c2fd8d91deb599aeb904fa47475bad1e0ff0aa9790b7b3ebe475930a029eec not found: ID does not exist"
Nov 04 00:36:37.349686 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.349654    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:36:37.354011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:37.353988    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ade4-544b6f4bcd-kq4qq"]
Nov 04 00:36:38.231073 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:38.231039    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" path="/var/lib/kubelet/pods/871e77b8-0d34-4fdc-aeac-1b110a9a1eba/volumes"
Nov 04 00:36:42.505513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.505478    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 04 00:36:42.505898 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.505672    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" podUID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" containerName="maas-api" containerID="cri-o://facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee" gracePeriod=30
Nov 04 00:36:42.801999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.801969    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 04 00:36:42.966613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.966577    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\" (UniqueName: \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\") pod \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") "
Nov 04 00:36:42.966770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.966630    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\") pod \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") "
Nov 04 00:36:42.966770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.966731    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\") pod \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") "
Nov 04 00:36:42.966770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.966764    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\") pod \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\" (UID: \"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f\") "
Nov 04 00:36:42.966996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.966969    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca" pod "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" (UID: "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:42.967119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.967102    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:42.968878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.968844    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs" pod "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" (UID: "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:42.968878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.968856    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5" pod "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" (UID: "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f"). InnerVolumeSpecName "kube-api-access-86zl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:42.968878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:42.968865    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls" pod "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" (UID: "cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:43.068502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.068469    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:43.068502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.068499    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:43.068704 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.068509    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zl5\" (UniqueName: \"kubernetes.io/projected/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f-kube-api-access-86zl5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:43.349710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.349611    6430 generic.go:356] "Generic (PLEG): container finished" podID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" containerID="facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee" exitCode=0
Nov 04 00:36:43.349710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.349681    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" event={"ID":"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f","Type":"ContainerDied","Data":"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"}
Nov 04 00:36:43.349710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.349708    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"
Nov 04 00:36:43.349978 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.349736    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm" event={"ID":"cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f","Type":"ContainerDied","Data":"b1765df54b9b93012d54679690f620bf2b2e3a83fc267ab1b9d9baaad1509f94"}
Nov 04 00:36:43.349978 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.349766    6430 scope.go:122] "RemoveContainer" containerID="facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"
Nov 04 00:36:43.360065 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.360042    6430 scope.go:122] "RemoveContainer" containerID="facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"
Nov 04 00:36:43.360363 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:43.360340    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee\": container with ID starting with facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee not found: ID does not exist" containerID="facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"
Nov 04 00:36:43.360449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.360372    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee"} err="failed to get container status \"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee\": rpc error: code = NotFound desc = could not find container \"facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee\": container with ID starting with facc599d34ef68c7161b3c2c479915a541a080c36cf6e535ab3243902b69aeee not found: ID does not exist"
Nov 04 00:36:43.375489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.375457    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 04 00:36:43.379719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:43.379698    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5738a9-58659c7c79-qffsm"]
Nov 04 00:36:44.234806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:44.234764    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" path="/var/lib/kubelet/pods/cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f/volumes"
Nov 04 00:36:44.250562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:44.244544    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:36:44.250562 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:44.244770    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy" containerID="cri-o://d550941ecc510536446683a3dc21b3397dff925aa90f8fa1471fc9c752b9cbf5" gracePeriod=30
Nov 04 00:36:49.391926 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.391885    6430 generic.go:356] "Generic (PLEG): container finished" podID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerID="d550941ecc510536446683a3dc21b3397dff925aa90f8fa1471fc9c752b9cbf5" exitCode=0
Nov 04 00:36:49.392441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.391942    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerDied","Data":"d550941ecc510536446683a3dc21b3397dff925aa90f8fa1471fc9c752b9cbf5"}
Nov 04 00:36:49.392441 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.392037    6430 scope.go:122] "RemoveContainer" containerID="db90c43dbea9eee24b8bd7d6f94d6bbd08092ac3b8c3351514487f58d2d8251c"
Nov 04 00:36:49.503660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.503633    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:36:49.522121 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522050    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522123    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522212    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522264    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522287    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522339    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522362    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522430    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:49.522507 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522440    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522519    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\") pod \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\" (UID: \"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6\") "
Nov 04 00:36:49.522809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522547    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:49.522809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522615    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:49.522809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522753    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:49.522809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522785    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:49.523088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522949    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.523088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522966    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.523088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522979    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.523088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.522993    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.523088 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.523006    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.524420 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.524383    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:36:49.524581 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.524562    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:49.524764 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.524744    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:49.524846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.524822    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx" pod "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" (UID: "ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6"). InnerVolumeSpecName "kube-api-access-g5zkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:49.624349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.624313    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zkx\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-kube-api-access-g5zkx\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.624349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.624346    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.624349 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.624355    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:49.624606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:49.624365    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:50.401126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:50.401091    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"
Nov 04 00:36:50.401644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:50.401123    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz" event={"ID":"ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6","Type":"ContainerDied","Data":"7b18cdd52e08ce25931cc98898c00ecc24252e0aa52f4ea0a013f391cd3ff14a"}
Nov 04 00:36:50.401644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:50.401201    6430 scope.go:122] "RemoveContainer" containerID="d550941ecc510536446683a3dc21b3397dff925aa90f8fa1471fc9c752b9cbf5"
Nov 04 00:36:50.423849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:50.423816    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:36:50.426169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:50.426145    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ade4-openshift-default-555fb6cd85-mkhgz"]
Nov 04 00:36:51.008110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:51.008071    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 04 00:36:51.757822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:51.757758    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main" containerID="cri-o://351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496" gracePeriod=45
Nov 04 00:36:52.014281 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.014216    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 04 00:36:52.063806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.063772    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.063976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.063852    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.064030 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.063969    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.064030 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064003    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.064157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064059    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.064157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064094    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\" (UniqueName: \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\") pod \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\" (UID: \"6d463f76-9ae4-4b76-848d-2ace5fecfb94\") "
Nov 04 00:36:52.064280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064183    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.064478 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064442    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.064591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064539    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.064591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064551    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.064591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.064559    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.066604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.066577    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:52.066746 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.066722    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "kube-api-access-26l9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:52.066793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.066743    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm" pod "6d463f76-9ae4-4b76-848d-2ace5fecfb94" (UID: "6d463f76-9ae4-4b76-848d-2ace5fecfb94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:36:52.165629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.165591    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.165629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.165630    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d463f76-9ae4-4b76-848d-2ace5fecfb94-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.165846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.165646    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26l9r\" (UniqueName: \"kubernetes.io/projected/6d463f76-9ae4-4b76-848d-2ace5fecfb94-kube-api-access-26l9r\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.165846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.165661    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d463f76-9ae4-4b76-848d-2ace5fecfb94-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:52.230523 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.230487    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" path="/var/lib/kubelet/pods/ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6/volumes"
Nov 04 00:36:52.414299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.414259    6430 generic.go:356] "Generic (PLEG): container finished" podID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerID="351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496" exitCode=0
Nov 04 00:36:52.414545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.414308    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"
Nov 04 00:36:52.414545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.414340    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" event={"ID":"6d463f76-9ae4-4b76-848d-2ace5fecfb94","Type":"ContainerDied","Data":"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"}
Nov 04 00:36:52.414545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.414416    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt" event={"ID":"6d463f76-9ae4-4b76-848d-2ace5fecfb94","Type":"ContainerDied","Data":"a6015c9dc9b575ec7920a79a744014c56841532de9152a1a2c5b99f95a4d0e99"}
Nov 04 00:36:52.414545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.414441    6430 scope.go:122] "RemoveContainer" containerID="351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"
Nov 04 00:36:52.424458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.424433    6430 scope.go:122] "RemoveContainer" containerID="351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"
Nov 04 00:36:52.424760 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:52.424736    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496\": container with ID starting with 351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496 not found: ID does not exist" containerID="351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"
Nov 04 00:36:52.424824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.424766    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496"} err="failed to get container status \"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496\": rpc error: code = NotFound desc = could not find container \"351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496\": container with ID starting with 351057fed3ed6839ac904be08c4af08691d3dfa152c98eccc1bc0ae63195d496 not found: ID does not exist"
Nov 04 00:36:52.434077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.434043    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 04 00:36:52.438170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:52.438143    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/simulator-w3-c60893-kserve-6b9c5bd76f-fpkmt"]
Nov 04 00:36:53.607598 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.607559    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 04 00:36:53.608027 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.607794    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" podUID="fd3d8d66-451b-45cd-a6f2-599698ce696a" containerName="maas-api" containerID="cri-o://eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051" gracePeriod=30
Nov 04 00:36:53.866135 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.866067    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 04 00:36:53.986765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.986731    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\") pod \"fd3d8d66-451b-45cd-a6f2-599698ce696a\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") "
Nov 04 00:36:53.986922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.986814    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\") pod \"fd3d8d66-451b-45cd-a6f2-599698ce696a\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") "
Nov 04 00:36:53.986922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.986854    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\" (UniqueName: \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\") pod \"fd3d8d66-451b-45cd-a6f2-599698ce696a\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") "
Nov 04 00:36:53.986922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.986906    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\") pod \"fd3d8d66-451b-45cd-a6f2-599698ce696a\" (UID: \"fd3d8d66-451b-45cd-a6f2-599698ce696a\") "
Nov 04 00:36:53.987223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.987194    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca" pod "fd3d8d66-451b-45cd-a6f2-599698ce696a" (UID: "fd3d8d66-451b-45cd-a6f2-599698ce696a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:53.988963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.988934    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs" pod "fd3d8d66-451b-45cd-a6f2-599698ce696a" (UID: "fd3d8d66-451b-45cd-a6f2-599698ce696a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:53.989113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.988993    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6" pod "fd3d8d66-451b-45cd-a6f2-599698ce696a" (UID: "fd3d8d66-451b-45cd-a6f2-599698ce696a"). InnerVolumeSpecName "kube-api-access-7bvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:53.989113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:53.989025    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls" pod "fd3d8d66-451b-45cd-a6f2-599698ce696a" (UID: "fd3d8d66-451b-45cd-a6f2-599698ce696a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:36:54.088575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.088539    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:54.088575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.088570    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3d8d66-451b-45cd-a6f2-599698ce696a-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:54.088575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.088581    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvt6\" (UniqueName: \"kubernetes.io/projected/fd3d8d66-451b-45cd-a6f2-599698ce696a-kube-api-access-7bvt6\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:54.088823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.088590    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3d8d66-451b-45cd-a6f2-599698ce696a-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:54.233638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.233546    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" path="/var/lib/kubelet/pods/6d463f76-9ae4-4b76-848d-2ace5fecfb94/volumes"
Nov 04 00:36:54.426543 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.426506    6430 generic.go:356] "Generic (PLEG): container finished" podID="fd3d8d66-451b-45cd-a6f2-599698ce696a" containerID="eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051" exitCode=0
Nov 04 00:36:54.426719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.426571    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"
Nov 04 00:36:54.426719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.426589    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" event={"ID":"fd3d8d66-451b-45cd-a6f2-599698ce696a","Type":"ContainerDied","Data":"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"}
Nov 04 00:36:54.426719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.426644    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd" event={"ID":"fd3d8d66-451b-45cd-a6f2-599698ce696a","Type":"ContainerDied","Data":"6781ea604f5999192721ef946ff5c6a77ff098392913a2f66b70dbc9943d0466"}
Nov 04 00:36:54.426719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.426662    6430 scope.go:122] "RemoveContainer" containerID="eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"
Nov 04 00:36:54.435977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.435955    6430 scope.go:122] "RemoveContainer" containerID="eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"
Nov 04 00:36:54.436256 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:54.436233    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051\": container with ID starting with eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051 not found: ID does not exist" containerID="eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"
Nov 04 00:36:54.436359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.436260    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051"} err="failed to get container status \"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051\": rpc error: code = NotFound desc = could not find container \"eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051\": container with ID starting with eae4c2dcfe8ef8e59932513e4a0d89a1c16d27f54a0e9ff3dc8d6459cef74051 not found: ID does not exist"
Nov 04 00:36:54.446660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.446637    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 04 00:36:54.450860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:54.450836    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b3bea7-7b5cf54695-6qlgd"]
Nov 04 00:36:55.515540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:55.515503    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:36:56.009329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.009284    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" podUID="2e782547-9b13-46d2-8298-ce8919181256" containerName="payload-processing" containerID="cri-o://5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6" gracePeriod=25
Nov 04 00:36:56.232608 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.232573    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3d8d66-451b-45cd-a6f2-599698ce696a" path="/var/lib/kubelet/pods/fd3d8d66-451b-45cd-a6f2-599698ce696a/volumes"
Nov 04 00:36:56.274555 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.274494    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 04 00:36:56.416747 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.416716    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\" (UniqueName: \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\") pod \"2e782547-9b13-46d2-8298-ce8919181256\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") "
Nov 04 00:36:56.416899 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.416838    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\") pod \"2e782547-9b13-46d2-8298-ce8919181256\" (UID: \"2e782547-9b13-46d2-8298-ce8919181256\") "
Nov 04 00:36:56.417211 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.417185    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume" pod "2e782547-9b13-46d2-8298-ce8919181256" (UID: "2e782547-9b13-46d2-8298-ce8919181256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:36:56.418963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.418935    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5" pod "2e782547-9b13-46d2-8298-ce8919181256" (UID: "2e782547-9b13-46d2-8298-ce8919181256"). InnerVolumeSpecName "kube-api-access-9z2n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:36:56.437537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.437513    6430 generic.go:356] "Generic (PLEG): container finished" podID="2e782547-9b13-46d2-8298-ce8919181256" containerID="5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6" exitCode=0
Nov 04 00:36:56.437651 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.437567    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"
Nov 04 00:36:56.437651 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.437601    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" event={"ID":"2e782547-9b13-46d2-8298-ce8919181256","Type":"ContainerDied","Data":"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"}
Nov 04 00:36:56.437771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.437660    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt" event={"ID":"2e782547-9b13-46d2-8298-ce8919181256","Type":"ContainerDied","Data":"a03c75fd09acd11514191b15a8189730f7b95bd0a994188f1083d439b53b1d3a"}
Nov 04 00:36:56.437771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.437684    6430 scope.go:122] "RemoveContainer" containerID="5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"
Nov 04 00:36:56.446966 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.446945    6430 scope.go:122] "RemoveContainer" containerID="5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"
Nov 04 00:36:56.447207 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:36:56.447188    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6\": container with ID starting with 5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6 not found: ID does not exist" containerID="5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"
Nov 04 00:36:56.447256 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.447217    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6"} err="failed to get container status \"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6\": rpc error: code = NotFound desc = could not find container \"5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6\": container with ID starting with 5e7faad756e945fff12521e24b92e2ae58521275bf46d091b3a8d709334ba9e6 not found: ID does not exist"
Nov 04 00:36:56.463700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.463680    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 04 00:36:56.468422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.468402    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b3bea7-6cd8b6696-gbjnt"]
Nov 04 00:36:56.518001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.517972    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2n5\" (UniqueName: \"kubernetes.io/projected/2e782547-9b13-46d2-8298-ce8919181256-kube-api-access-9z2n5\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:56.518001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:56.517999    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e782547-9b13-46d2-8298-ce8919181256-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:36:58.229900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.229859    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e782547-9b13-46d2-8298-ce8919181256" path="/var/lib/kubelet/pods/2e782547-9b13-46d2-8298-ce8919181256/volumes"
Nov 04 00:36:58.835412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835355    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:36:58.835807 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835792    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835808    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835820    6430 cpu_manager.go:414] "containerMap: removing container" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835825    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835836    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" containerName="maas-api"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835841    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" containerName="maas-api"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835847    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835853    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835867    6430 cpu_manager.go:414] "containerMap: removing container" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing"
Nov 04 00:36:58.835878 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835872    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835882    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835887    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835894    6430 cpu_manager.go:414] "containerMap: removing container" podUID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835898    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835904    6430 cpu_manager.go:414] "containerMap: removing container" podUID="fd3d8d66-451b-45cd-a6f2-599698ce696a" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835908    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3d8d66-451b-45cd-a6f2-599698ce696a" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835919    6430 cpu_manager.go:414] "containerMap: removing container" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835923    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835931    6430 cpu_manager.go:414] "containerMap: removing container" podUID="9cd8c821-c06d-45e8-a643-4a4825404f30" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835938    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd8c821-c06d-45e8-a643-4a4825404f30" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835946    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ff8bc764-815f-4618-8d83-cba9e0d57def" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835951    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8bc764-815f-4618-8d83-cba9e0d57def" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835981    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835986    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835993    6430 cpu_manager.go:414] "containerMap: removing container" podUID="2e782547-9b13-46d2-8298-ce8919181256" containerName="payload-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.835998    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e782547-9b13-46d2-8298-ce8919181256" containerName="payload-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836005    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836013    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836024    6430 cpu_manager.go:414] "containerMap: removing container" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836029    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836035    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1d51fc39-822d-4d36-af2f-5dd386f57a78" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836039    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d51fc39-822d-4d36-af2f-5dd386f57a78" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836048    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836053    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836060    6430 cpu_manager.go:414] "containerMap: removing container" podUID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836064    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" containerName="maas-api"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836070    6430 cpu_manager.go:414] "containerMap: removing container" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836075    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836083    6430 cpu_manager.go:414] "containerMap: removing container" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836090    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836100    6430 cpu_manager.go:414] "containerMap: removing container" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836105    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836113    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836117    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836124    6430 cpu_manager.go:414] "containerMap: removing container" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836129    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836136    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836141    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836151    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836156    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836163    6430 cpu_manager.go:414] "containerMap: removing container" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836171    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing"
Nov 04 00:36:58.836161 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836180    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836186    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836197    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836202    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836207    6430 cpu_manager.go:414] "containerMap: removing container" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836212    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836223    6430 cpu_manager.go:414] "containerMap: removing container" podUID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836227    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836234    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836240    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836251    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836257    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836343    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c967b0-d1e9-4e4e-a4af-b613ab839c7d" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836351    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836359    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8008f1c5-c313-4781-8347-e155daa11c4b" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836365    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="605c8350-f8cd-4208-8185-bfcbaa38856f" containerName="payload-pre-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836372    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1903fd85-6808-4007-8cec-1aa4a137456c" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836380    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836410    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1226101c-28de-4eed-a854-c6db2a7b8540" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836419    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fdf076-ffef-4a57-9630-4f6e0f5a8ebe" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836426    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba151b3b-b90e-4c46-a5d7-7b2464b0f9a6" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836435    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b8ff2b-c457-417c-a7ed-76fba58bd015" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836443    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7b1139-91b7-44e9-a78f-e8f4d46e83ed" containerName="main"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836449    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836455    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c13f4ca-345b-4842-bd79-f341aa0d5efa" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836461    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836466    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836475    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="004957b7-0d92-42cc-a846-876ac4dfd3cf" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836485    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d463f76-9ae4-4b76-848d-2ace5fecfb94" containerName="main"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836494    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbb7e6f-eec4-41cf-9ad4-e7879d0d512f" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836504    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3d8d66-451b-45cd-a6f2-599698ce696a" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836510    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcda01ba-ca83-4671-8e3c-b2ffc559f992" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836516    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd8c821-c06d-45e8-a643-4a4825404f30" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836522    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8bc764-815f-4618-8d83-cba9e0d57def" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836527    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836533    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e782547-9b13-46d2-8298-ce8919181256" containerName="payload-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836538    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25d18f2-34e3-4912-8f9b-0cffcaec49da" containerName="istio-proxy"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836543    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="689807c7-203c-4dec-82fa-a2a8925be4ae" containerName="main"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836548    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d51fc39-822d-4d36-af2f-5dd386f57a78" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836553    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8600c0-1c62-4155-9bb0-4a13f50bb8df" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836558    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="871e77b8-0d34-4fdc-aeac-1b110a9a1eba" containerName="maas-api"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836563    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="499d212b-e9d7-424e-92cf-07b3f681208f" containerName="payload-pre-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836568    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fea092-5031-4ce0-b79f-42501c49a619" containerName="payload-pre-processing"
Nov 04 00:36:58.837558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.836573    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="209517d0-0b9c-4657-8d07-1c74ab5505ed" containerName="istio-proxy"
Nov 04 00:36:58.840868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.840845    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.851528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.851498    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\""
Nov 04 00:36:58.853332 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.853307    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:36:58.941707 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941674    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.941865 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941719    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.941865 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941813    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkl8\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.941865 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941851    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.941974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941946    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.942012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.941972    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.942072 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.942059    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.942123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.942104    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:58.942176 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:58.942139    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043005    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043051    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043089    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkl8\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043113    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043369 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043244    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043369 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043279    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043369 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043355    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043587 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043382    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043587 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043448    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043594    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043693 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043647    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043719    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.043837 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.043805    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.044093 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.044076    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.045475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.045455    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.045593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.045562    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.056090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.056062    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkl8\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.056090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.056074    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.154913 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.154849    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:36:59.436834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.436756    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:36:59.440158 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:36:59.440130    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317b235a_8732_4254_8dc0_4422afd8db6d.slice/crio-c1f3c2f02473201329c64792d806efff1b644b930cb46d1ae0d1b0c4436094b9 WatchSource:0}: Error finding container c1f3c2f02473201329c64792d806efff1b644b930cb46d1ae0d1b0c4436094b9: Status 404 returned error can't find the container with id c1f3c2f02473201329c64792d806efff1b644b930cb46d1ae0d1b0c4436094b9
Nov 04 00:36:59.442536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.442501    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:36:59.442605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.442570    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:36:59.442605 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.442598    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:36:59.465128 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:36:59.465100    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" event={"ID":"317b235a-8732-4254-8dc0-4422afd8db6d","Type":"ContainerStarted","Data":"c1f3c2f02473201329c64792d806efff1b644b930cb46d1ae0d1b0c4436094b9"}
Nov 04 00:37:00.471434 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:00.471359    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" event={"ID":"317b235a-8732-4254-8dc0-4422afd8db6d","Type":"ContainerStarted","Data":"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"}
Nov 04 00:37:00.499655 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:00.499604    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" podStartSLOduration=2.499591699 podStartE2EDuration="2.499591699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:37:00.497563111 +0000 UTC m=+4036.388625047" watchObservedRunningTime="2026-11-04 00:37:00.499591699 +0000 UTC m=+4036.390653628"
Nov 04 00:37:01.155145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.155097    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:01.160401 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.160359    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:01.284184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.284140    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:37:01.284413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.284357    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" containerID="cri-o://3fcdb26b471397c0ff409ca148b5b08b6a4d0e7aa6da1d2223e2fbbf300d4f39" gracePeriod=30
Nov 04 00:37:01.478025 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.477939    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:01.478923 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.478901    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:01.741822 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.741734    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 04 00:37:01.741986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:01.741955    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" podUID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" containerName="maas-api" containerID="cri-o://38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0" gracePeriod=30
Nov 04 00:37:02.003479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.003375    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 04 00:37:02.007900 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.007871    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:37:02.078180 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.078144    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\" (UniqueName: \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\") pod \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") "
Nov 04 00:37:02.078347 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.078334    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\") pod \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") "
Nov 04 00:37:02.078417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.078381    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\") pod \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") "
Nov 04 00:37:02.078502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.078444    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\") pod \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\" (UID: \"7078d1f9-107d-44cc-b2d9-5b064ed72fc4\") "
Nov 04 00:37:02.078816 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.078784    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca" pod "7078d1f9-107d-44cc-b2d9-5b064ed72fc4" (UID: "7078d1f9-107d-44cc-b2d9-5b064ed72fc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:02.080332 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.080301    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf" pod "7078d1f9-107d-44cc-b2d9-5b064ed72fc4" (UID: "7078d1f9-107d-44cc-b2d9-5b064ed72fc4"). InnerVolumeSpecName "kube-api-access-xk8bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:02.080484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.080452    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs" pod "7078d1f9-107d-44cc-b2d9-5b064ed72fc4" (UID: "7078d1f9-107d-44cc-b2d9-5b064ed72fc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:02.080583 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.080514    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls" pod "7078d1f9-107d-44cc-b2d9-5b064ed72fc4" (UID: "7078d1f9-107d-44cc-b2d9-5b064ed72fc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:02.179845 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.179809    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8bf\" (UniqueName: \"kubernetes.io/projected/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-kube-api-access-xk8bf\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:02.179845 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.179842    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:02.179845 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.179853    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:02.180083 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.179862    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7078d1f9-107d-44cc-b2d9-5b064ed72fc4-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:02.483996 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.483958    6430 generic.go:356] "Generic (PLEG): container finished" podID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" containerID="38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0" exitCode=0
Nov 04 00:37:02.484467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.484010    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"
Nov 04 00:37:02.484467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.484038    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" event={"ID":"7078d1f9-107d-44cc-b2d9-5b064ed72fc4","Type":"ContainerDied","Data":"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"}
Nov 04 00:37:02.484467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.484091    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh" event={"ID":"7078d1f9-107d-44cc-b2d9-5b064ed72fc4","Type":"ContainerDied","Data":"12e3377a88e4f40d8240bc5a7fb181f5bd3c84742d54d6303008c5528998e4ca"}
Nov 04 00:37:02.484467 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.484113    6430 scope.go:122] "RemoveContainer" containerID="38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"
Nov 04 00:37:02.484662 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.484474    6430 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" secret="" err="secret \"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\" not found"
Nov 04 00:37:02.493379 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.493360    6430 scope.go:122] "RemoveContainer" containerID="38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"
Nov 04 00:37:02.493665 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:02.493645    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0\": container with ID starting with 38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0 not found: ID does not exist" containerID="38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"
Nov 04 00:37:02.493736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.493676    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0"} err="failed to get container status \"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0\": rpc error: code = NotFound desc = could not find container \"38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0\": container with ID starting with 38ed648f57c790d402ea3d139050e812a92b4e7a08d243c6da3d853651ece1e0 not found: ID does not exist"
Nov 04 00:37:02.503417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.503370    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 04 00:37:02.506877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:02.506855    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c60893-698545b546-g26bh"]
Nov 04 00:37:03.490638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:03.490591    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" podUID="317b235a-8732-4254-8dc0-4422afd8db6d" containerName="istio-proxy" containerID="cri-o://f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483" gracePeriod=30
Nov 04 00:37:04.231992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:04.231946    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" path="/var/lib/kubelet/pods/7078d1f9-107d-44cc-b2d9-5b064ed72fc4/volumes"
Nov 04 00:37:04.831046 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:04.831000    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:37:05.919475 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:05.919438    6430 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:37:05.919993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:05.919497    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:37:06.507715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.507675    6430 generic.go:356] "Generic (PLEG): container finished" podID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerID="3fcdb26b471397c0ff409ca148b5b08b6a4d0e7aa6da1d2223e2fbbf300d4f39" exitCode=0
Nov 04 00:37:06.507893 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.507749    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" event={"ID":"3508bcc2-87c3-4f2a-b2da-8aa6c4192299","Type":"ContainerDied","Data":"3fcdb26b471397c0ff409ca148b5b08b6a4d0e7aa6da1d2223e2fbbf300d4f39"}
Nov 04 00:37:06.535786 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.535758    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:37:06.624712 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624635    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624712 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624701    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624731    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624780    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624806    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624853    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.624939 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624932    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.625274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.624957    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\") pod \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\" (UID: \"3508bcc2-87c3-4f2a-b2da-8aa6c4192299\") "
Nov 04 00:37:06.625274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625049    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:06.625274 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625075    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:06.625490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625277    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:06.625490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625295    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:06.625490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625361    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.625490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625382    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.625490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.625460    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:06.627134 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.627105    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:06.627252 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.627140    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:06.627252 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.627174    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:06.627252 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.627209    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm" pod "3508bcc2-87c3-4f2a-b2da-8aa6c4192299" (UID: "3508bcc2-87c3-4f2a-b2da-8aa6c4192299"). InnerVolumeSpecName "kube-api-access-njxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:06.726100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726071    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726095    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726107    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726116    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxbm\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-kube-api-access-njxbm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726126    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726134    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:06.726264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:06.726143    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3508bcc2-87c3-4f2a-b2da-8aa6c4192299-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:07.370765 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.370729    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:37:07.514297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.514252    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw" event={"ID":"3508bcc2-87c3-4f2a-b2da-8aa6c4192299","Type":"ContainerDied","Data":"d759ac527592a8c6a9c708ba51a4e4b59a68dd9ddb844a90d13bf9abb8671c39"}
Nov 04 00:37:07.514297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.514289    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"
Nov 04 00:37:07.514577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.514319    6430 scope.go:122] "RemoveContainer" containerID="3fcdb26b471397c0ff409ca148b5b08b6a4d0e7aa6da1d2223e2fbbf300d4f39"
Nov 04 00:37:07.539141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.539116    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:37:07.543663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:07.543641    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-r42pw"]
Nov 04 00:37:08.230644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.230615    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" path="/var/lib/kubelet/pods/3508bcc2-87c3-4f2a-b2da-8aa6c4192299/volumes"
Nov 04 00:37:08.741573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.741549    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:08.842195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842160    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842210    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842284    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842312    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842337    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842366    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842426    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842463    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842495    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\") pod \"317b235a-8732-4254-8dc0-4422afd8db6d\" (UID: \"317b235a-8732-4254-8dc0-4422afd8db6d\") "
Nov 04 00:37:08.842835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842546    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.842835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842672    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.842835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.842683    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.843116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.843028    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.843116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.843044    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.843116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.843053    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.843116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.843091    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.843116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.843053    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:08.844597 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.844577    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "kube-api-access-4jkl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:08.844879 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.844865    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:08.845013 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.844978    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:08.845089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.845026    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token" pod "317b235a-8732-4254-8dc0-4422afd8db6d" (UID: "317b235a-8732-4254-8dc0-4422afd8db6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:08.944509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944466    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317b235a-8732-4254-8dc0-4422afd8db6d-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.944509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944501    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.944509 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944511    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317b235a-8732-4254-8dc0-4422afd8db6d-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.944770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944520    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkl8\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-kube-api-access-4jkl8\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.944770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944530    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317b235a-8732-4254-8dc0-4422afd8db6d-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:08.944770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:08.944538    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317b235a-8732-4254-8dc0-4422afd8db6d-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:09.526102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.526067    6430 generic.go:356] "Generic (PLEG): container finished" podID="317b235a-8732-4254-8dc0-4422afd8db6d" containerID="f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483" exitCode=0
Nov 04 00:37:09.526324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.526111    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"
Nov 04 00:37:09.526324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.526143    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" event={"ID":"317b235a-8732-4254-8dc0-4422afd8db6d","Type":"ContainerDied","Data":"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"}
Nov 04 00:37:09.526324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.526193    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps" event={"ID":"317b235a-8732-4254-8dc0-4422afd8db6d","Type":"ContainerDied","Data":"c1f3c2f02473201329c64792d806efff1b644b930cb46d1ae0d1b0c4436094b9"}
Nov 04 00:37:09.526324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.526220    6430 scope.go:122] "RemoveContainer" containerID="f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"
Nov 04 00:37:09.536299 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.536282    6430 scope.go:122] "RemoveContainer" containerID="f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"
Nov 04 00:37:09.536584 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:09.536565    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483\": container with ID starting with f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483 not found: ID does not exist" containerID="f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"
Nov 04 00:37:09.536653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.536594    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483"} err="failed to get container status \"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483\": rpc error: code = NotFound desc = could not find container \"f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483\": container with ID starting with f7b3845045cd35272800c0d4a69465ee681878b1a53ff61f188df71664e74483 not found: ID does not exist"
Nov 04 00:37:09.551074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.551052    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:37:09.554757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:09.554737    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-hhsps"]
Nov 04 00:37:10.230043 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:10.230002    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317b235a-8732-4254-8dc0-4422afd8db6d" path="/var/lib/kubelet/pods/317b235a-8732-4254-8dc0-4422afd8db6d/volumes"
Nov 04 00:37:13.057138 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057099    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:13.057542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057507    6430 cpu_manager.go:414] "containerMap: removing container" podUID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" containerName="maas-api"
Nov 04 00:37:13.057542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057519    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" containerName="maas-api"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057561    6430 cpu_manager.go:414] "containerMap: removing container" podUID="317b235a-8732-4254-8dc0-4422afd8db6d" containerName="istio-proxy"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057568    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317b235a-8732-4254-8dc0-4422afd8db6d" containerName="istio-proxy"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057577    6430 cpu_manager.go:414] "containerMap: removing container" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057583    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057602    6430 cpu_manager.go:414] "containerMap: removing container" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:37:13.057619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057607    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46731542-c799-4992-89c3-221d1bd1c18a" containerName="istio-proxy"
Nov 04 00:37:13.057831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057666    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fee7e4-0040-404c-8305-fd7cbbb1c5cd" containerName="istio-proxy"
Nov 04 00:37:13.057831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057678    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="7078d1f9-107d-44cc-b2d9-5b064ed72fc4" containerName="maas-api"
Nov 04 00:37:13.057831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057686    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="317b235a-8732-4254-8dc0-4422afd8db6d" containerName="istio-proxy"
Nov 04 00:37:13.057831 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.057692    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="3508bcc2-87c3-4f2a-b2da-8aa6c4192299" containerName="istio-proxy"
Nov 04 00:37:13.062225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.062205    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.067974 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.067946    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c60893-openshift-default-dockercfg-hwjgp\""
Nov 04 00:37:13.076291 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.076264    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:13.187536 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187499    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187545    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187613    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187667    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187703    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187724    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187824    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.187886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187852    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.188003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.187918    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq464\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289137 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289103    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289149    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289176    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289205    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq464\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289229    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289253    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289277    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289318    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289345    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289570    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289772 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289613    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.289991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.289971    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.290062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.290025    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.290305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.290284    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.291723 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.291702    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.292041 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.292017    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.302329 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.302306    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.302519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.302501    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq464\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.377922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.377864    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:13.661459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.661431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:13.664001 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:37:13.663975    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f24093_b1fa_4b96_a4bb_03fdc942115e.slice/crio-9feec35c8b6d8c417744cdd5b3190e8e97e6eda349cee921d8b7eac20fe5a314 WatchSource:0}: Error finding container 9feec35c8b6d8c417744cdd5b3190e8e97e6eda349cee921d8b7eac20fe5a314: Status 404 returned error can't find the container with id 9feec35c8b6d8c417744cdd5b3190e8e97e6eda349cee921d8b7eac20fe5a314
Nov 04 00:37:13.666110 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.666077    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:37:13.666191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.666145    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:37:13.666191 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:13.666174    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:37:14.552346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:14.552301    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" event={"ID":"44f24093-b1fa-4b96-a4bb-03fdc942115e","Type":"ContainerStarted","Data":"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"}
Nov 04 00:37:14.552767 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:14.552364    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" event={"ID":"44f24093-b1fa-4b96-a4bb-03fdc942115e","Type":"ContainerStarted","Data":"9feec35c8b6d8c417744cdd5b3190e8e97e6eda349cee921d8b7eac20fe5a314"}
Nov 04 00:37:14.580008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:14.579953    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" podStartSLOduration=1.5799399589999998 podStartE2EDuration="1.579939959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:37:14.579556285 +0000 UTC m=+4050.470618219" watchObservedRunningTime="2026-11-04 00:37:14.579939959 +0000 UTC m=+4050.471001894"
Nov 04 00:37:15.378884 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:15.378848    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:15.380337 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:15.380311    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 04 00:37:15.380480 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:15.380380    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 04 00:37:16.378948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:16.378908    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 04 00:37:16.379359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:16.378976    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 04 00:37:16.480968 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:16.480930    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:37:16.847413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:16.845953    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:17.378496 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:17.378460    6430 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 04 00:37:17.378676 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:17.378515    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 04 00:37:17.567967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:17.567920    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy" containerID="cri-o://f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee" gracePeriod=30
Nov 04 00:37:19.205242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:19.205203    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:37:19.206079 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:19.206039    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" containerID="cri-o://8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b" gracePeriod=30
Nov 04 00:37:20.850102 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:20.850064    6430 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 04 00:37:20.850508 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:20.850124    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:37:24.490424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.490383    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:37:24.530163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530089    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530138    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530195    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530366 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530226    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530360    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530502 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530420    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530615 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530532    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:24.530615 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530559    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530615 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530610    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.530787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530606    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:24.530787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530657    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:24.530787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530760    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:24.530967 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530951    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:24.531051 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.530701    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\") pod \"73298ba3-d100-4e00-8708-cc9de5588a9e\" (UID: \"73298ba3-d100-4e00-8708-cc9de5588a9e\") "
Nov 04 00:37:24.531408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.531322    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.531408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.531344    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73298ba3-d100-4e00-8708-cc9de5588a9e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.531408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.531355    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.531408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.531364    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.531408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.531372    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.532431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.532401    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:24.532870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.532841    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:24.532962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.532898    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "kube-api-access-xlb4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:24.533113 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.533097    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy" pod "73298ba3-d100-4e00-8708-cc9de5588a9e" (UID: "73298ba3-d100-4e00-8708-cc9de5588a9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:24.602479 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.602438    6430 generic.go:356] "Generic (PLEG): container finished" podID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerID="8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b" exitCode=0
Nov 04 00:37:24.602660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.602487    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"
Nov 04 00:37:24.602660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.602520    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" event={"ID":"73298ba3-d100-4e00-8708-cc9de5588a9e","Type":"ContainerDied","Data":"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"}
Nov 04 00:37:24.602660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.602581    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7" event={"ID":"73298ba3-d100-4e00-8708-cc9de5588a9e","Type":"ContainerDied","Data":"2eae1ace97a8d19113bbd5e21fb56761c732b7ce7f78027fd9ef0d70a5c6d3c9"}
Nov 04 00:37:24.602660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.602605    6430 scope.go:122] "RemoveContainer" containerID="8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"
Nov 04 00:37:24.613642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.613622    6430 scope.go:122] "RemoveContainer" containerID="8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"
Nov 04 00:37:24.613947 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:24.613928    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b\": container with ID starting with 8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b not found: ID does not exist" containerID="8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"
Nov 04 00:37:24.614022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.613958    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b"} err="failed to get container status \"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b\": rpc error: code = NotFound desc = could not find container \"8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b\": container with ID starting with 8f011c986250fb0b4a4ca0a92d98237ca9435335e893cc68a7b2e3b46ba6599b not found: ID does not exist"
Nov 04 00:37:24.628793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.628759    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:37:24.632824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.632798    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.632934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.632827    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.632934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.632842    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.632934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.632855    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlb4w\" (UniqueName: \"kubernetes.io/projected/73298ba3-d100-4e00-8708-cc9de5588a9e-kube-api-access-xlb4w\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:24.635267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.635241    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-7rll7"]
Nov 04 00:37:24.740984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.740942    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 04 00:37:24.751975 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.751940    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 04 00:37:24.759757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:24.759707    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 04 00:37:25.203019 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.202993    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:25.238418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238313    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238418 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238358    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238453    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238511    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238539    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238576    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238596    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.238653 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238646    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.239047 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.238675    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\") pod \"44f24093-b1fa-4b96-a4bb-03fdc942115e\" (UID: \"44f24093-b1fa-4b96-a4bb-03fdc942115e\") "
Nov 04 00:37:25.239205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.239122    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.239205 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.239189    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.239315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.239239    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:37:25.239416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.239372    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.239482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.239453    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.240978 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.240947    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:37:25.241577 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.241552    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:25.241668 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.241639    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "kube-api-access-gq464". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:25.241833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.241804    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token" pod "44f24093-b1fa-4b96-a4bb-03fdc942115e" (UID: "44f24093-b1fa-4b96-a4bb-03fdc942115e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:25.340449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340409    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f24093-b1fa-4b96-a4bb-03fdc942115e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340443    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340455    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340465    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq464\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-kube-api-access-gq464\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340475    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340483    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340492    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340501    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f24093-b1fa-4b96-a4bb-03fdc942115e-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.340702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.340509    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f24093-b1fa-4b96-a4bb-03fdc942115e-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:25.608748 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.608709    6430 generic.go:356] "Generic (PLEG): container finished" podID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerID="f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee" exitCode=0
Nov 04 00:37:25.609182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.608750    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" event={"ID":"44f24093-b1fa-4b96-a4bb-03fdc942115e","Type":"ContainerDied","Data":"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"}
Nov 04 00:37:25.609182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.608813    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7" event={"ID":"44f24093-b1fa-4b96-a4bb-03fdc942115e","Type":"ContainerDied","Data":"9feec35c8b6d8c417744cdd5b3190e8e97e6eda349cee921d8b7eac20fe5a314"}
Nov 04 00:37:25.609182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.608825    6430 scope.go:122] "RemoveContainer" containerID="f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"
Nov 04 00:37:25.609182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.608760    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"
Nov 04 00:37:25.618347 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.618331    6430 scope.go:122] "RemoveContainer" containerID="f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"
Nov 04 00:37:25.618613 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:25.618593    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee\": container with ID starting with f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee not found: ID does not exist" containerID="f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"
Nov 04 00:37:25.618685 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.618623    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee"} err="failed to get container status \"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee\": rpc error: code = NotFound desc = could not find container \"f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee\": container with ID starting with f4a2e411244b1395ef2f6c29bb5e41940d212b270e007c43b36059b624273dee not found: ID does not exist"
Nov 04 00:37:25.634228 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.634190    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:25.638327 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:25.638300    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-fj9g7"]
Nov 04 00:37:26.230231 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:26.230196    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" path="/var/lib/kubelet/pods/44f24093-b1fa-4b96-a4bb-03fdc942115e/volumes"
Nov 04 00:37:26.230725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:26.230703    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" path="/var/lib/kubelet/pods/73298ba3-d100-4e00-8708-cc9de5588a9e/volumes"
Nov 04 00:37:26.604074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:26.604026    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:37:34.829694 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:34.829654    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:37:39.762165 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:39.762101    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main" containerID="cri-o://f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d" gracePeriod=45
Nov 04 00:37:39.773145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:39.773089    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main" containerID="cri-o://daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32" gracePeriod=45
Nov 04 00:37:39.783799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:39.783762    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main" containerID="cri-o://61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3" gracePeriod=45
Nov 04 00:37:40.053751 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.053716    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 04 00:37:40.075482 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.075032    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 04 00:37:40.094424 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.094379    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 04 00:37:40.182591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182554    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.182787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182600    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.182787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182645    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\" (UniqueName: \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.182787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182685    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.182787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182723    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.182787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182755    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182790    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182834    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\" (UniqueName: \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182859    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182897    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\" (UniqueName: \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\") pod \"df47b886-ddb6-497a-b569-85fed4ca9c36\" (UID: \"df47b886-ddb6-497a-b569-85fed4ca9c36\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182901    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182926    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.183126 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.182958    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.183541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183427    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.183541 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183372    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.183659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183544    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.183659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183618    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.183922 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183899    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.184068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.183996    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.184068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184032    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\" (UniqueName: \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.184234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184074    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.184234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184130    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\") pod \"0b371dfa-eaf6-4e45-a606-d641d9645dab\" (UID: \"0b371dfa-eaf6-4e45-a606-d641d9645dab\") "
Nov 04 00:37:40.184234 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184183    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\") pod \"46bf7f64-b200-40bd-8373-05fbbc3dc847\" (UID: \"46bf7f64-b200-40bd-8373-05fbbc3dc847\") "
Nov 04 00:37:40.184622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184580    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.184726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184639    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.184726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184657    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.184726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184671    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.184726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184684    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.185054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184781    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.185054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.184889    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.185054 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.185016    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.185225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.185097    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.185557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.185530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:40.186702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.186636    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:40.186702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.186669    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.188159 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.188138    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt" pod "df47b886-ddb6-497a-b569-85fed4ca9c36" (UID: "df47b886-ddb6-497a-b569-85fed4ca9c36"). InnerVolumeSpecName "kube-api-access-xsmgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:40.188280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.188170    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "kube-api-access-z7j7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:40.188354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.188281    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:37:40.188354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.188311    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "kube-api-access-49m5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:37:40.188354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.188331    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm" pod "46bf7f64-b200-40bd-8373-05fbbc3dc847" (UID: "46bf7f64-b200-40bd-8373-05fbbc3dc847"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.189551 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.189530    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm" pod "0b371dfa-eaf6-4e45-a606-d641d9645dab" (UID: "0b371dfa-eaf6-4e45-a606-d641d9645dab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285746    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df47b886-ddb6-497a-b569-85fed4ca9c36-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285778    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df47b886-ddb6-497a-b569-85fed4ca9c36-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285787    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b371dfa-eaf6-4e45-a606-d641d9645dab-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285798    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49m5g\" (UniqueName: \"kubernetes.io/projected/46bf7f64-b200-40bd-8373-05fbbc3dc847-kube-api-access-49m5g\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285809    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmgt\" (UniqueName: \"kubernetes.io/projected/df47b886-ddb6-497a-b569-85fed4ca9c36-kube-api-access-xsmgt\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285818    6430 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46bf7f64-b200-40bd-8373-05fbbc3dc847-tls-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285827    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.285829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285836    6430 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-dshm\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.286220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285844    6430 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-tmp-dir\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.286220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285852    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7j7l\" (UniqueName: \"kubernetes.io/projected/0b371dfa-eaf6-4e45-a606-d641d9645dab-kube-api-access-z7j7l\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.286220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285860    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.286220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285869    6430 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b371dfa-eaf6-4e45-a606-d641d9645dab-home\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.286220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.285878    6430 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46bf7f64-b200-40bd-8373-05fbbc3dc847-model-cache\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:37:40.684200 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.684160    6430 generic.go:356] "Generic (PLEG): container finished" podID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerID="daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32" exitCode=0
Nov 04 00:37:40.684436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.684212    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"
Nov 04 00:37:40.684436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.684252    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" event={"ID":"df47b886-ddb6-497a-b569-85fed4ca9c36","Type":"ContainerDied","Data":"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"}
Nov 04 00:37:40.684436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.684300    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7" event={"ID":"df47b886-ddb6-497a-b569-85fed4ca9c36","Type":"ContainerDied","Data":"1c493aed4fcf5a9ee33318870e08cbb2a1aef5c08cf274b5ea3f61ac922e331c"}
Nov 04 00:37:40.684436 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.684318    6430 scope.go:122] "RemoveContainer" containerID="daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"
Nov 04 00:37:40.685650 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.685622    6430 generic.go:356] "Generic (PLEG): container finished" podID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerID="f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d" exitCode=0
Nov 04 00:37:40.685788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.685671    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"
Nov 04 00:37:40.685788 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.685668    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" event={"ID":"0b371dfa-eaf6-4e45-a606-d641d9645dab","Type":"ContainerDied","Data":"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"}
Nov 04 00:37:40.685929 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.685794    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9" event={"ID":"0b371dfa-eaf6-4e45-a606-d641d9645dab","Type":"ContainerDied","Data":"315961442fb17fc7b09e028ac248d22eeead146fdd13f5dfcc628bbed9340705"}
Nov 04 00:37:40.687376 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.687356    6430 generic.go:356] "Generic (PLEG): container finished" podID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerID="61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3" exitCode=0
Nov 04 00:37:40.687517 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.687407    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" event={"ID":"46bf7f64-b200-40bd-8373-05fbbc3dc847","Type":"ContainerDied","Data":"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"}
Nov 04 00:37:40.687517 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.687417    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"
Nov 04 00:37:40.687517 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.687439    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm" event={"ID":"46bf7f64-b200-40bd-8373-05fbbc3dc847","Type":"ContainerDied","Data":"e8a2ab49b5128611c2e8856fdbc0ca2e91dbbcb4fe58d10771daf4aaf53284b4"}
Nov 04 00:37:40.694343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.694324    6430 scope.go:122] "RemoveContainer" containerID="daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"
Nov 04 00:37:40.694634 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:40.694615    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32\": container with ID starting with daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32 not found: ID does not exist" containerID="daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"
Nov 04 00:37:40.694688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.694642    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32"} err="failed to get container status \"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32\": rpc error: code = NotFound desc = could not find container \"daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32\": container with ID starting with daa9a47510c478bc876b3d4c7110f61df1789872ca553d29ea468ee3c1e36f32 not found: ID does not exist"
Nov 04 00:37:40.694688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.694664    6430 scope.go:122] "RemoveContainer" containerID="f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"
Nov 04 00:37:40.702460 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.702440    6430 scope.go:122] "RemoveContainer" containerID="f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"
Nov 04 00:37:40.702756 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:40.702722    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d\": container with ID starting with f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d not found: ID does not exist" containerID="f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"
Nov 04 00:37:40.702830 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.702764    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d"} err="failed to get container status \"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d\": rpc error: code = NotFound desc = could not find container \"f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d\": container with ID starting with f5f57f997287b0c38d4d3a2b246a9b447a013aaa61fc8e8da8bd67c128859d2d not found: ID does not exist"
Nov 04 00:37:40.702830 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.702780    6430 scope.go:122] "RemoveContainer" containerID="61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"
Nov 04 00:37:40.710648 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.710558    6430 scope.go:122] "RemoveContainer" containerID="61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"
Nov 04 00:37:40.710904 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:37:40.710873    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3\": container with ID starting with 61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3 not found: ID does not exist" containerID="61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"
Nov 04 00:37:40.711015 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.710915    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3"} err="failed to get container status \"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3\": rpc error: code = NotFound desc = could not find container \"61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3\": container with ID starting with 61d789691bd82a835590d2f9a82b8b1fe3bc393356b07c9c142b9bb65dba88f3 not found: ID does not exist"
Nov 04 00:37:40.713090 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.713065    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 04 00:37:40.716770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.716746    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/premium-w4-80f8ba-kserve-788fdd7d99-jbjp7"]
Nov 04 00:37:40.729501 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.729473    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 04 00:37:40.733285 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.733262    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/distinct-2-w4-80f8ba-kserve-8b48df9d9-xr4z9"]
Nov 04 00:37:40.746166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.746141    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 04 00:37:40.750173 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:40.750151    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-80f8ba/simulator-w4-80f8ba-kserve-69cb8f598b-7dmbm"]
Nov 04 00:37:42.229242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:42.229202    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" path="/var/lib/kubelet/pods/0b371dfa-eaf6-4e45-a606-d641d9645dab/volumes"
Nov 04 00:37:42.229672 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:42.229645    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" path="/var/lib/kubelet/pods/46bf7f64-b200-40bd-8373-05fbbc3dc847/volumes"
Nov 04 00:37:42.230020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:42.230006    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" path="/var/lib/kubelet/pods/df47b886-ddb6-497a-b569-85fed4ca9c36/volumes"
Nov 04 00:37:45.164794 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:37:45.164755    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:38:04.462852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:04.462815    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 00:38:04.830641 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:38:04.830605    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:38:05.551869 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:05.551827    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 00:38:05.751808 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:05.751776    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"
Nov 04 00:38:16.375003 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:16.374973    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 00:38:28.269149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:28.269112    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 00:38:34.600843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.600800    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:38:34.601226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601162    6430 cpu_manager.go:414] "containerMap: removing container" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main"
Nov 04 00:38:34.601226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601173    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main"
Nov 04 00:38:34.601226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601226    6430 cpu_manager.go:414] "containerMap: removing container" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601231    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601243    6430 cpu_manager.go:414] "containerMap: removing container" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601248    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601260    6430 cpu_manager.go:414] "containerMap: removing container" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601264    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601273    6430 cpu_manager.go:414] "containerMap: removing container" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601278    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601338    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="73298ba3-d100-4e00-8708-cc9de5588a9e" containerName="istio-proxy"
Nov 04 00:38:34.601345 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601346    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f24093-b1fa-4b96-a4bb-03fdc942115e" containerName="istio-proxy"
Nov 04 00:38:34.601674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601355    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bf7f64-b200-40bd-8373-05fbbc3dc847" containerName="main"
Nov 04 00:38:34.601674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601361    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b371dfa-eaf6-4e45-a606-d641d9645dab" containerName="main"
Nov 04 00:38:34.601674 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.601369    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="df47b886-ddb6-497a-b569-85fed4ca9c36" containerName="main"
Nov 04 00:38:34.604544 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.604523    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.613212 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.613189    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-849709d8-openshift-default-dockercfg-lf89d\""
Nov 04 00:38:34.617210 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.617179    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:38:34.771355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771323    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771415    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771490    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771563 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771544    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771567    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslqp\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771587    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771601    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771734 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771675    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.771925 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.771752    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.830467 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:38:34.830434    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:38:34.872999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.872928    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.872999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.872965    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.872999 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.872982    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873003    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslqp\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873024    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873046    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873090    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873129    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873171    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873441    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873541    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873611 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873601    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873755 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873632    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.873826 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.873808    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.875187 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.875166    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.875585 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.875566    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.886226 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.886201    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.886466 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.886449    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslqp\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\") pod \"e2e-ait-849709d8-openshift-default-76bb4876d7-thshx\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") " pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:34.919272 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:34.919252    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:35.198174 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.198143    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:38:35.200752 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:38:35.200724    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513086dd_be9e_43c8_9f2e_f977eb6d0071.slice/crio-3c21b6de3a191487e8cc471fae137bd056e3ffea6f85824a3b038b6dc02fab9f WatchSource:0}: Error finding container 3c21b6de3a191487e8cc471fae137bd056e3ffea6f85824a3b038b6dc02fab9f: Status 404 returned error can't find the container with id 3c21b6de3a191487e8cc471fae137bd056e3ffea6f85824a3b038b6dc02fab9f
Nov 04 00:38:35.202848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.202818    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:35.202952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.202879    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:35.202952 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.202905    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:35.934042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.934007    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerStarted","Data":"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"}
Nov 04 00:38:35.934476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.934062    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerStarted","Data":"3c21b6de3a191487e8cc471fae137bd056e3ffea6f85824a3b038b6dc02fab9f"}
Nov 04 00:38:35.963237 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:35.963190    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podStartSLOduration=1.963173701 podStartE2EDuration="1.963173701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:38:35.960225848 +0000 UTC m=+4131.851287824" watchObservedRunningTime="2026-11-04 00:38:35.963173701 +0000 UTC m=+4131.854235626"
Nov 04 00:38:36.919759 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:36.919722    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:36.924725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:36.924699    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:36.938143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:36.938113    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:36.938817 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:36.938797    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:38.589921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.589889    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:38:38.593716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.593694    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.607542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.602656    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-849709d8-serving-cert\""
Nov 04 00:38:38.608938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.608847    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.609168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.609148    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgvc\" (UniqueName: \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.609351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.609326    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.609351 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.609075    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:38:38.609583 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.609420    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.637022 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.636992    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:38:38.641020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.640998    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.647860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.647837    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:38:38.649378 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.649356    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-849709d8\""
Nov 04 00:38:38.661075 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.661052    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:38:38.664575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.664553    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.674119 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.674099    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:38:38.683238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.683206    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-849709d8-dockercfg-kdskm\""
Nov 04 00:38:38.710089 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710063    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.710220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710099    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgvc\" (UniqueName: \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.710220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710132    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.710220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710185    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.710220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710216    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmhk\" (UniqueName: \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.710415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710316    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.710415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710374    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.710497 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710433    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.710709 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.710692    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.712867 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.712836    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.712962 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.712850    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.722837 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.722817    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgvc\" (UniqueName: \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\") pod \"maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.812702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.811806    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.812702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.811872    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.812702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.811978    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmhk\" (UniqueName: \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.812702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.812060    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.812702 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.812666    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.813005 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.812790    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.825408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.825372    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmhk\" (UniqueName: \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\") pod \"payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") " pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.825572 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.825520    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\") pod \"payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:38.912063 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.912002    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:38.955307 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.955282    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:38.981921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:38.981692    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:39.186487 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.186456    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:38:39.188578 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:38:39.188545    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5413805a_d3bb_46a5_b7ab_9652e565e6ef.slice/crio-83a9cbf0c65e8995d740b62f75624bc6b4571d34913f68543eb66ae339380c62 WatchSource:0}: Error finding container 83a9cbf0c65e8995d740b62f75624bc6b4571d34913f68543eb66ae339380c62: Status 404 returned error can't find the container with id 83a9cbf0c65e8995d740b62f75624bc6b4571d34913f68543eb66ae339380c62
Nov 04 00:38:39.190659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.190629    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:38:39.250239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.250207    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:38:39.254098 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:38:39.254073    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db65293_1869_4f15_9e38_98d6417ecd00.slice/crio-f833049ecadf47cd9e488b48bcb743f46bffda489aeaaf8eb519d9a8f2585a6a WatchSource:0}: Error finding container f833049ecadf47cd9e488b48bcb743f46bffda489aeaaf8eb519d9a8f2585a6a: Status 404 returned error can't find the container with id f833049ecadf47cd9e488b48bcb743f46bffda489aeaaf8eb519d9a8f2585a6a
Nov 04 00:38:39.270800 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.270776    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:38:39.286583 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:38:39.286558    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521132b7_117f_4628_9059_df67b8a270f9.slice/crio-1fc4ff9a2a4c9a38f222c891d216bf5709d66996ab26c02c671f776cb1e66bdb WatchSource:0}: Error finding container 1fc4ff9a2a4c9a38f222c891d216bf5709d66996ab26c02c671f776cb1e66bdb: Status 404 returned error can't find the container with id 1fc4ff9a2a4c9a38f222c891d216bf5709d66996ab26c02c671f776cb1e66bdb
Nov 04 00:38:39.952305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.952269    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" event={"ID":"5413805a-d3bb-46a5-b7ab-9652e565e6ef","Type":"ContainerStarted","Data":"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"}
Nov 04 00:38:39.952719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.952333    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" event={"ID":"5413805a-d3bb-46a5-b7ab-9652e565e6ef","Type":"ContainerStarted","Data":"83a9cbf0c65e8995d740b62f75624bc6b4571d34913f68543eb66ae339380c62"}
Nov 04 00:38:39.952719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.952365    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:39.953832 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.953805    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" event={"ID":"6db65293-1869-4f15-9e38-98d6417ecd00","Type":"ContainerStarted","Data":"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"}
Nov 04 00:38:39.953961 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.953854    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" event={"ID":"6db65293-1869-4f15-9e38-98d6417ecd00","Type":"ContainerStarted","Data":"f833049ecadf47cd9e488b48bcb743f46bffda489aeaaf8eb519d9a8f2585a6a"}
Nov 04 00:38:39.953961 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.953902    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:39.955258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.955237    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" event={"ID":"521132b7-117f-4628-9059-df67b8a270f9","Type":"ContainerStarted","Data":"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"}
Nov 04 00:38:39.955353 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.955270    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" event={"ID":"521132b7-117f-4628-9059-df67b8a270f9","Type":"ContainerStarted","Data":"1fc4ff9a2a4c9a38f222c891d216bf5709d66996ab26c02c671f776cb1e66bdb"}
Nov 04 00:38:39.955353 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.955299    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:39.974133 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.974088    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" podStartSLOduration=1.325404341 podStartE2EDuration="1.974075648s" totalImagesPullingTime="648.671307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:38:39.973341259 +0000 UTC m=+4135.864403194" watchObservedRunningTime="2026-11-04 00:38:39.974075648 +0000 UTC m=+4135.865137574"
Nov 04 00:38:39.994355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:39.994312    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" podStartSLOduration=1.994299991 podStartE2EDuration="1.994299991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:38:39.993175686 +0000 UTC m=+4135.884237622" watchObservedRunningTime="2026-11-04 00:38:39.994299991 +0000 UTC m=+4135.885361920"
Nov 04 00:38:40.013100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:40.013062    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" podStartSLOduration=2.013048708 podStartE2EDuration="2.013048708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:38:40.012681648 +0000 UTC m=+4135.903743781" watchObservedRunningTime="2026-11-04 00:38:40.013048708 +0000 UTC m=+4135.904110643"
Nov 04 00:38:40.498673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:40.498638    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:38:45.965692 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:45.965660    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:38:50.011076 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.010969    6430 generic.go:356] "Generic (PLEG): container finished" podID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerID="b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807" exitCode=0
Nov 04 00:38:50.011076 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.011027    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerDied","Data":"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"}
Nov 04 00:38:50.011726 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.011523    6430 scope.go:122] "RemoveContainer" containerID="b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"
Nov 04 00:38:50.012732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.012699    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:50.012834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.012768    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:50.012834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.012799    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:38:50.963944 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.963915    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:38:50.964169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:50.963981    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:38:51.042613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:51.042572    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerStarted","Data":"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"}
Nov 04 00:38:51.920886 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:51.920849    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:51.931430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:51.927504    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 04 00:38:51.931430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:51.927565    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 04 00:38:51.938860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:51.938824    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:52.926412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:52.925756    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 04 00:38:52.926412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:52.925819    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 04 00:38:53.923889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:53.923799    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 04 00:38:53.924409 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:53.924215    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 04 00:38:54.924414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:54.921872    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 04 00:38:54.924414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:54.921962    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 04 00:38:55.920143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:55.920096    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 04 00:38:55.920380 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:55.920153    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 04 00:38:56.923713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:56.923682    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:38:56.924627 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:38:56.924605    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:39:03.247742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:03.247154    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:39:03.260603 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:03.260561    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:39:03.822973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:03.822936    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:39:03.823203 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:03.823161    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" podUID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" containerName="maas-api" containerID="cri-o://a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13" gracePeriod=30
Nov 04 00:39:04.071613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.071589    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:39:04.109524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.109438    6430 generic.go:356] "Generic (PLEG): container finished" podID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" containerID="a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13" exitCode=0
Nov 04 00:39:04.109524 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.109487    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"
Nov 04 00:39:04.109761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.109532    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" event={"ID":"5413805a-d3bb-46a5-b7ab-9652e565e6ef","Type":"ContainerDied","Data":"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"}
Nov 04 00:39:04.109761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.109595    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr" event={"ID":"5413805a-d3bb-46a5-b7ab-9652e565e6ef","Type":"ContainerDied","Data":"83a9cbf0c65e8995d740b62f75624bc6b4571d34913f68543eb66ae339380c62"}
Nov 04 00:39:04.109761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.109621    6430 scope.go:122] "RemoveContainer" containerID="a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"
Nov 04 00:39:04.118218 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.118201    6430 scope.go:122] "RemoveContainer" containerID="a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"
Nov 04 00:39:04.118524 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:04.118497    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13\": container with ID starting with a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13 not found: ID does not exist" containerID="a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"
Nov 04 00:39:04.118596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.118535    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13"} err="failed to get container status \"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13\": rpc error: code = NotFound desc = could not find container \"a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13\": container with ID starting with a365ba23713d85f35f7ad70afd063be9fb7c6d4f6d6392927cc8a0fe5400da13 not found: ID does not exist"
Nov 04 00:39:04.166801 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.166776    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\") pod \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") "
Nov 04 00:39:04.166914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.166851    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\") pod \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") "
Nov 04 00:39:04.166914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.166872    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\" (UniqueName: \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\") pod \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") "
Nov 04 00:39:04.166914 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.166900    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\") pod \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\" (UID: \"5413805a-d3bb-46a5-b7ab-9652e565e6ef\") "
Nov 04 00:39:04.167198 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.167172    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca" pod "5413805a-d3bb-46a5-b7ab-9652e565e6ef" (UID: "5413805a-d3bb-46a5-b7ab-9652e565e6ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:39:04.168973 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.168938    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs" pod "5413805a-d3bb-46a5-b7ab-9652e565e6ef" (UID: "5413805a-d3bb-46a5-b7ab-9652e565e6ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:04.169062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.169021    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls" pod "5413805a-d3bb-46a5-b7ab-9652e565e6ef" (UID: "5413805a-d3bb-46a5-b7ab-9652e565e6ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:04.169062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.169036    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc" pod "5413805a-d3bb-46a5-b7ab-9652e565e6ef" (UID: "5413805a-d3bb-46a5-b7ab-9652e565e6ef"). InnerVolumeSpecName "kube-api-access-8kgvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:04.267846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.267818    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:04.267846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.267843    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgvc\" (UniqueName: \"kubernetes.io/projected/5413805a-d3bb-46a5-b7ab-9652e565e6ef-kube-api-access-8kgvc\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:04.268199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.267853    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5413805a-d3bb-46a5-b7ab-9652e565e6ef-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:04.268199 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.267863    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5413805a-d3bb-46a5-b7ab-9652e565e6ef-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:04.427214 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.427135    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:39:04.430880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:04.430851    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-849709d8-56bf9d49cf-qlmnr"]
Nov 04 00:39:04.830013 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:04.829977    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:39:05.390185 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:05.390153    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:39:06.230375 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:06.230337    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" path="/var/lib/kubelet/pods/5413805a-d3bb-46a5-b7ab-9652e565e6ef/volumes"
Nov 04 00:39:08.247793 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.247753    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" podUID="6db65293-1869-4f15-9e38-98d6417ecd00" containerName="payload-processing" containerID="cri-o://19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd" gracePeriod=25
Nov 04 00:39:08.261504 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.261477    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" podUID="521132b7-117f-4628-9059-df67b8a270f9" containerName="payload-pre-processing" containerID="cri-o://0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6" gracePeriod=25
Nov 04 00:39:08.528116 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.528091    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:39:08.533238 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.533215    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:39:08.604426 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604369    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\") pod \"521132b7-117f-4628-9059-df67b8a270f9\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") "
Nov 04 00:39:08.604622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604454    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\") pod \"521132b7-117f-4628-9059-df67b8a270f9\" (UID: \"521132b7-117f-4628-9059-df67b8a270f9\") "
Nov 04 00:39:08.604622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604501    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\") pod \"6db65293-1869-4f15-9e38-98d6417ecd00\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") "
Nov 04 00:39:08.604622 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604536    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\" (UniqueName: \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\") pod \"6db65293-1869-4f15-9e38-98d6417ecd00\" (UID: \"6db65293-1869-4f15-9e38-98d6417ecd00\") "
Nov 04 00:39:08.604849 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604825    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume" pod "521132b7-117f-4628-9059-df67b8a270f9" (UID: "521132b7-117f-4628-9059-df67b8a270f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:39:08.604937 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.604912    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume" pod "6db65293-1869-4f15-9e38-98d6417ecd00" (UID: "6db65293-1869-4f15-9e38-98d6417ecd00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:39:08.606603 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.606584    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j" pod "521132b7-117f-4628-9059-df67b8a270f9" (UID: "521132b7-117f-4628-9059-df67b8a270f9"). InnerVolumeSpecName "kube-api-access-8g95j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:08.606700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.606681    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk" pod "6db65293-1869-4f15-9e38-98d6417ecd00" (UID: "6db65293-1869-4f15-9e38-98d6417ecd00"). InnerVolumeSpecName "kube-api-access-bcmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:08.705833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.705803    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521132b7-117f-4628-9059-df67b8a270f9-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:08.705833 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.705829    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/521132b7-117f-4628-9059-df67b8a270f9-kube-api-access-8g95j\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:08.706016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.705839    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db65293-1869-4f15-9e38-98d6417ecd00-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:08.706016 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:08.705850    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcmhk\" (UniqueName: \"kubernetes.io/projected/6db65293-1869-4f15-9e38-98d6417ecd00-kube-api-access-bcmhk\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:09.134115 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.134075    6430 generic.go:356] "Generic (PLEG): container finished" podID="521132b7-117f-4628-9059-df67b8a270f9" containerID="0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6" exitCode=0
Nov 04 00:39:09.134304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.134127    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"
Nov 04 00:39:09.134304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.134159    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" event={"ID":"521132b7-117f-4628-9059-df67b8a270f9","Type":"ContainerDied","Data":"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"}
Nov 04 00:39:09.134304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.134215    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb" event={"ID":"521132b7-117f-4628-9059-df67b8a270f9","Type":"ContainerDied","Data":"1fc4ff9a2a4c9a38f222c891d216bf5709d66996ab26c02c671f776cb1e66bdb"}
Nov 04 00:39:09.134304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.134233    6430 scope.go:122] "RemoveContainer" containerID="0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"
Nov 04 00:39:09.135583 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.135563    6430 generic.go:356] "Generic (PLEG): container finished" podID="6db65293-1869-4f15-9e38-98d6417ecd00" containerID="19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd" exitCode=0
Nov 04 00:39:09.135715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.135598    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" event={"ID":"6db65293-1869-4f15-9e38-98d6417ecd00","Type":"ContainerDied","Data":"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"}
Nov 04 00:39:09.135715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.135608    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"
Nov 04 00:39:09.135715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.135629    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4" event={"ID":"6db65293-1869-4f15-9e38-98d6417ecd00","Type":"ContainerDied","Data":"f833049ecadf47cd9e488b48bcb743f46bffda489aeaaf8eb519d9a8f2585a6a"}
Nov 04 00:39:09.143103 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.143086    6430 scope.go:122] "RemoveContainer" containerID="0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"
Nov 04 00:39:09.143495 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:09.143471    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6\": container with ID starting with 0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6 not found: ID does not exist" containerID="0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"
Nov 04 00:39:09.143624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.143504    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6"} err="failed to get container status \"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6\": rpc error: code = NotFound desc = could not find container \"0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6\": container with ID starting with 0a205d6bb69b92cb5a65dc9907eba52debfc948a2f7597adcec37bcb50d140d6 not found: ID does not exist"
Nov 04 00:39:09.143624 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.143526    6430 scope.go:122] "RemoveContainer" containerID="19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"
Nov 04 00:39:09.151612 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.151560    6430 scope.go:122] "RemoveContainer" containerID="19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"
Nov 04 00:39:09.151808 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:09.151782    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd\": container with ID starting with 19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd not found: ID does not exist" containerID="19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"
Nov 04 00:39:09.151883 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.151815    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd"} err="failed to get container status \"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd\": rpc error: code = NotFound desc = could not find container \"19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd\": container with ID starting with 19442010f1d6aa2e643eaad3deaa56182ab00c20dfb70f525c9cedf0030b1acd not found: ID does not exist"
Nov 04 00:39:09.162147 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.162124    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:39:09.165661 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.165639    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-849709d8-754b4765ff-6pmjb"]
Nov 04 00:39:09.179034 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.179010    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:39:09.180668 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.180647    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-849709d8-7fd598db6d-gnfc4"]
Nov 04 00:39:09.587815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.587777    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:39:09.588202 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:09.587983    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" containerID="cri-o://573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5" gracePeriod=30
Nov 04 00:39:10.230354 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:10.230317    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521132b7-117f-4628-9059-df67b8a270f9" path="/var/lib/kubelet/pods/521132b7-117f-4628-9059-df67b8a270f9/volumes"
Nov 04 00:39:10.230736 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:10.230720    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db65293-1869-4f15-9e38-98d6417ecd00" path="/var/lib/kubelet/pods/6db65293-1869-4f15-9e38-98d6417ecd00/volumes"
Nov 04 00:39:11.925870 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:11.925831    6430 patch_prober.go:29] interesting pod/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 04 00:39:11.926320 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:11.925893    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 04 00:39:13.112068 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112009    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:13.112498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112477    6430 cpu_manager.go:414] "containerMap: removing container" podUID="6db65293-1869-4f15-9e38-98d6417ecd00" containerName="payload-processing"
Nov 04 00:39:13.112498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112492    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db65293-1869-4f15-9e38-98d6417ecd00" containerName="payload-processing"
Nov 04 00:39:13.112596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112502    6430 cpu_manager.go:414] "containerMap: removing container" podUID="521132b7-117f-4628-9059-df67b8a270f9" containerName="payload-pre-processing"
Nov 04 00:39:13.112596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112507    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521132b7-117f-4628-9059-df67b8a270f9" containerName="payload-pre-processing"
Nov 04 00:39:13.112596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112522    6430 cpu_manager.go:414] "containerMap: removing container" podUID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" containerName="maas-api"
Nov 04 00:39:13.112596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112527    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" containerName="maas-api"
Nov 04 00:39:13.112744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112615    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db65293-1869-4f15-9e38-98d6417ecd00" containerName="payload-processing"
Nov 04 00:39:13.112744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112626    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="521132b7-117f-4628-9059-df67b8a270f9" containerName="payload-pre-processing"
Nov 04 00:39:13.112744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.112633    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="5413805a-d3bb-46a5-b7ab-9652e565e6ef" containerName="maas-api"
Nov 04 00:39:13.115782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.115761    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.125549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.125521    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ww46r\""
Nov 04 00:39:13.125549 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.125536    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 04 00:39:13.140433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.127610    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 04 00:39:13.140433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.129453    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:13.140433 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.129628    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 04 00:39:13.250760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.250729    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.250927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.250782    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zk9\" (UniqueName: \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.250927 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.250848    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.352292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.352254    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zk9\" (UniqueName: \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.352484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.352353    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.352575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.352502    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.354821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.354796    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.354934 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.354824    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.366239 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.366181    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zk9\" (UniqueName: \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\") pod \"maas-controller-9cc8d5ff6-6xqkw\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") " pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.438059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.438031    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:13.714260 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:13.714234    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:13.716768 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:39:13.716741    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23c108b_6f63_45bd_8c5c_2270a73646ff.slice/crio-8176c3dfebb9528af2121a00b72eb3870fc7cdfbb607d5482afe57b2fa893333 WatchSource:0}: Error finding container 8176c3dfebb9528af2121a00b72eb3870fc7cdfbb607d5482afe57b2fa893333: Status 404 returned error can't find the container with id 8176c3dfebb9528af2121a00b72eb3870fc7cdfbb607d5482afe57b2fa893333
Nov 04 00:39:14.162761 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.162727    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" event={"ID":"d23c108b-6f63-45bd-8c5c-2270a73646ff","Type":"ContainerStarted","Data":"8176c3dfebb9528af2121a00b72eb3870fc7cdfbb607d5482afe57b2fa893333"}
Nov 04 00:39:14.645179 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:14.645150    6430 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513086dd_be9e_43c8_9f2e_f977eb6d0071.slice/crio-conmon-573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5.scope\": RecentStats: unable to find data in memory cache]"
Nov 04 00:39:14.839197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.839172    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:39:14.967106 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967023    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967126    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967171    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967211    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967254 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967245    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967282    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967339    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967377    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967447    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\") pod \"513086dd-be9e-43c8-9f2e-f977eb6d0071\" (UID: \"513086dd-be9e-43c8-9f2e-f977eb6d0071\") "
Nov 04 00:39:14.967750 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967597    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:39:14.967812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967767    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513086dd-be9e-43c8-9f2e-f977eb6d0071-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:14.967883 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967856    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:39:14.967953 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.967889    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:39:14.968037 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.968017    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:39:14.968092 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.968045    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:39:14.969727 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.969698    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "kube-api-access-bslqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:14.970216 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.970195    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:39:14.970318 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.970187    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:39:14.971520 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:14.971495    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token" pod "513086dd-be9e-43c8-9f2e-f977eb6d0071" (UID: "513086dd-be9e-43c8-9f2e-f977eb6d0071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:15.068601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068570    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068597    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068606    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068615    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068624    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068632    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068640    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513086dd-be9e-43c8-9f2e-f977eb6d0071-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.068821 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.068650    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslqp\" (UniqueName: \"kubernetes.io/projected/513086dd-be9e-43c8-9f2e-f977eb6d0071-kube-api-access-bslqp\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:15.170067 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.170031    6430 generic.go:356] "Generic (PLEG): container finished" podID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerID="573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5" exitCode=0
Nov 04 00:39:15.170566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.170082    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"
Nov 04 00:39:15.170566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.170120    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerDied","Data":"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"}
Nov 04 00:39:15.170566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.170227    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx" event={"ID":"513086dd-be9e-43c8-9f2e-f977eb6d0071","Type":"ContainerDied","Data":"3c21b6de3a191487e8cc471fae137bd056e3ffea6f85824a3b038b6dc02fab9f"}
Nov 04 00:39:15.170566 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.170260    6430 scope.go:122] "RemoveContainer" containerID="573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"
Nov 04 00:39:15.191715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.191689    6430 scope.go:122] "RemoveContainer" containerID="b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"
Nov 04 00:39:15.197123 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.197101    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:39:15.201177 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.201156    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-849709d8-openshift-default-76bb4876d7-thshx"]
Nov 04 00:39:15.202293 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.202277    6430 scope.go:122] "RemoveContainer" containerID="573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"
Nov 04 00:39:15.202708 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:15.202684    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5\": container with ID starting with 573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5 not found: ID does not exist" containerID="573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"
Nov 04 00:39:15.202809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.202714    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5"} err="failed to get container status \"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5\": rpc error: code = NotFound desc = could not find container \"573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5\": container with ID starting with 573b985266057a5263a03224330135bc168c87e28f4ccfcd6d43f992f06ef0d5 not found: ID does not exist"
Nov 04 00:39:15.202809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.202731    6430 scope.go:122] "RemoveContainer" containerID="b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"
Nov 04 00:39:15.203587 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:15.203559    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807\": container with ID starting with b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807 not found: ID does not exist" containerID="b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"
Nov 04 00:39:15.203700 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:15.203598    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807"} err="failed to get container status \"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807\": rpc error: code = NotFound desc = could not find container \"b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807\": container with ID starting with b1b653b0eba0da2433798f95a5ac16c2c8a4d4b584263419dacfc2f90e985807 not found: ID does not exist"
Nov 04 00:39:16.230043 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:16.230001    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" path="/var/lib/kubelet/pods/513086dd-be9e-43c8-9f2e-f977eb6d0071/volumes"
Nov 04 00:39:17.181503 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:17.181402    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" event={"ID":"d23c108b-6f63-45bd-8c5c-2270a73646ff","Type":"ContainerStarted","Data":"61d6306d0475fbe2ad7567f5c604740499e13bef8c742272e9870b7d2fb2237c"}
Nov 04 00:39:17.181689 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:17.181515    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:17.205920 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:17.205866    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" podStartSLOduration=1.100198617 podStartE2EDuration="4.205853534s" totalImagesPullingTime="3.105654917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:17.20484724 +0000 UTC m=+4173.095909178" watchObservedRunningTime="2026-11-04 00:39:17.205853534 +0000 UTC m=+4173.096915469"
Nov 04 00:39:28.188207 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:28.188175    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:34.830637 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:39:34.830601    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:39:43.977938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.977899    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"]
Nov 04 00:39:43.978331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978283    6430 cpu_manager.go:414] "containerMap: removing container" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.978331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978294    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.978331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978310    6430 cpu_manager.go:414] "containerMap: removing container" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.978331 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978315    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.978505 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978374    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.978505 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.978401    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="513086dd-be9e-43c8-9f2e-f977eb6d0071" containerName="istio-proxy"
Nov 04 00:39:43.981497 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.981470    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:43.994153 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.994131    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sh6t7\""
Nov 04 00:39:43.998029 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:43.998004    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"]
Nov 04 00:39:44.000880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.000854    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"]
Nov 04 00:39:44.004798 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.004778    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"]
Nov 04 00:39:44.004917 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.004899    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.007985 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.007966    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.021587 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.021563    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"]
Nov 04 00:39:44.022754 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.022728    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"]
Nov 04 00:39:44.027316 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027295    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.027614 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027583    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.027814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027788    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.027814 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027804    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"]
Nov 04 00:39:44.028026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027839    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027964    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028026 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.027991    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028160 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.028036    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkzx\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-kube-api-access-jpkzx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028160 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.028076    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028160 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.028099    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9de502-2a33-42ea-996d-f96b86c57a3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.028160 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.028133    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.046046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.046021    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"]
Nov 04 00:39:44.129078 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129046    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129083    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129223 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129101    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.129315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129290    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129344    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.129408 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129368    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129418    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.129488 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129467    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwqm\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-kube-api-access-mcwqm\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.129521 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129489    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5dt\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-kube-api-access-cx5dt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129558 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129549    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129593 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129571    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.129629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129600    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129664 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129633    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129663    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035f969-cba1-40f0-904d-ee65b610b96c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.129701 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129694    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.129805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129727    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkzx\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-kube-api-access-jpkzx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129805 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129791    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129918 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129794    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129918 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129833    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce493a-180f-47a6-a03e-74f3db88976f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.129918 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129888    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.129918 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129904    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.130074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.129968    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035f969-cba1-40f0-904d-ee65b610b96c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130007    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130032    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9de502-2a33-42ea-996d-f96b86c57a3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130074 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130061    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130098    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqvp\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-kube-api-access-mfqvp\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130170    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130200    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.130283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130229    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130292    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130310    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130338    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130366    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130463    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130473    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130516    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.130553 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130550    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.130860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130579    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.130860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130647    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.130860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130675    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.130860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.130686    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9de502-2a33-42ea-996d-f96b86c57a3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.132290 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.132267    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.132417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.132336    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.142011 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.141983    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.142638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.142613    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkzx\" (UniqueName: \"kubernetes.io/projected/eb9de502-2a33-42ea-996d-f96b86c57a3f-kube-api-access-jpkzx\") pod \"maas-default-gateway-openshift-default-778b7b5c47-9t5h4\" (UID: \"eb9de502-2a33-42ea-996d-f96b86c57a3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.231143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231074    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231143 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231116    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwqm\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-kube-api-access-mcwqm\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.231323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231142    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5dt\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-kube-api-access-cx5dt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231166    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231190    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.231323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231218    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035f969-cba1-40f0-904d-ee65b610b96c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231323 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231237    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231414    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231450    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce493a-180f-47a6-a03e-74f3db88976f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231480    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231506    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035f969-cba1-40f0-904d-ee65b610b96c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231539    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231543    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231569    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqvp\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-kube-api-access-mfqvp\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231597    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.231642 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231624    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231657    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231686    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231720    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231738    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231763    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231789    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231819    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231854    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231886    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231923    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231950    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.231981    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232014    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232034    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8035f969-cba1-40f0-904d-ee65b610b96c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.232152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232065    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232172    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232211    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce493a-180f-47a6-a03e-74f3db88976f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232336    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232349    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232608    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232608    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232858    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232858    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.233014 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.232924    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.233447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.233056    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.233447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.233083    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.234404 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.234341    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.234610 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.234585    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.234687 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.234632    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8035f969-cba1-40f0-904d-ee65b610b96c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.235169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.235147    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8035f969-cba1-40f0-904d-ee65b610b96c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.235278 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.235202    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.235498 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.235478    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.245449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245137    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5dt\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-kube-api-access-cx5dt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.245449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245157    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwqm\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-kube-api-access-mcwqm\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.245449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245356    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqvp\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-kube-api-access-mfqvp\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.245449 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245358    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b017e-aceb-4f7d-8470-4af3187aea6b-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-x42sk\" (UID: \"fb4b017e-aceb-4f7d-8470-4af3187aea6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.245866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245801    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce493a-180f-47a6-a03e-74f3db88976f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-tcztg\" (UID: \"c2ce493a-180f-47a6-a03e-74f3db88976f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.245866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.245808    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8035f969-cba1-40f0-904d-ee65b610b96c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-sjtz5\" (UID: \"8035f969-cba1-40f0-904d-ee65b610b96c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.295439 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.295413    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:44.323350 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.323330    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:44.335909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.335886    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:44.347647 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.347612    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:44.575917 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.575886    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"]
Nov 04 00:39:44.577546 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:39:44.577520    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9de502_2a33_42ea_996d_f96b86c57a3f.slice/crio-9054324776dd3d6ad5ce4739bf23648841b4dc8d77d09140d44b36cd6a8c9832 WatchSource:0}: Error finding container 9054324776dd3d6ad5ce4739bf23648841b4dc8d77d09140d44b36cd6a8c9832: Status 404 returned error can't find the container with id 9054324776dd3d6ad5ce4739bf23648841b4dc8d77d09140d44b36cd6a8c9832
Nov 04 00:39:44.579733 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.579703    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.579839 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.579775    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.579839 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.579819    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.614866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.614838    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"]
Nov 04 00:39:44.616632 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:39:44.616608    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ce493a_180f_47a6_a03e_74f3db88976f.slice/crio-a09e3eec6a17be093052c93a981955d2bfcf26c503e2af055e895ef458efd02f WatchSource:0}: Error finding container a09e3eec6a17be093052c93a981955d2bfcf26c503e2af055e895ef458efd02f: Status 404 returned error can't find the container with id a09e3eec6a17be093052c93a981955d2bfcf26c503e2af055e895ef458efd02f
Nov 04 00:39:44.618731 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.618700    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.618860 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.618771    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.618935 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.618897    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.647356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.647333    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"]
Nov 04 00:39:44.650477 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:39:44.650450    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8035f969_cba1_40f0_904d_ee65b610b96c.slice/crio-fdf98b90c6974faa53c9c8f7bd247e043c0a3f53c3dfa560d2cb7cafe3eca1c0 WatchSource:0}: Error finding container fdf98b90c6974faa53c9c8f7bd247e043c0a3f53c3dfa560d2cb7cafe3eca1c0: Status 404 returned error can't find the container with id fdf98b90c6974faa53c9c8f7bd247e043c0a3f53c3dfa560d2cb7cafe3eca1c0
Nov 04 00:39:44.653245 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.653220    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"]
Nov 04 00:39:44.656272 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.656238    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.656361 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.656314    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.656463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.656357    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.660688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.659643    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.660688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.659700    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.660688 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.659726    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:39:44.957246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.957150    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:44.957513 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:44.957456    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" podUID="d23c108b-6f63-45bd-8c5c-2270a73646ff" containerName="manager" containerID="cri-o://61d6306d0475fbe2ad7567f5c604740499e13bef8c742272e9870b7d2fb2237c" gracePeriod=30
Nov 04 00:39:45.559422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.532897    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" event={"ID":"eb9de502-2a33-42ea-996d-f96b86c57a3f","Type":"ContainerStarted","Data":"d5025f6a5ddbaf2f29be220057de2c13fb708a382d783a84f9ae766ae17825fd"}
Nov 04 00:39:45.559422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.532975    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" event={"ID":"eb9de502-2a33-42ea-996d-f96b86c57a3f","Type":"ContainerStarted","Data":"9054324776dd3d6ad5ce4739bf23648841b4dc8d77d09140d44b36cd6a8c9832"}
Nov 04 00:39:45.594415 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.583713    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podStartSLOduration=2.58369266 podStartE2EDuration="2.58369266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:45.581894964 +0000 UTC m=+4201.472956940" watchObservedRunningTime="2026-11-04 00:39:45.58369266 +0000 UTC m=+4201.474754596"
Nov 04 00:39:45.651079 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.651037    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" event={"ID":"8035f969-cba1-40f0-904d-ee65b610b96c","Type":"ContainerStarted","Data":"64aef635a5f918af4963fa2db127adbf5341ba39ed4c609e9a053d54cab6b7f9"}
Nov 04 00:39:45.672311 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.672278    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" event={"ID":"8035f969-cba1-40f0-904d-ee65b610b96c","Type":"ContainerStarted","Data":"fdf98b90c6974faa53c9c8f7bd247e043c0a3f53c3dfa560d2cb7cafe3eca1c0"}
Nov 04 00:39:45.689654 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.689590    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podStartSLOduration=2.689571058 podStartE2EDuration="2.689571058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:45.689006396 +0000 UTC m=+4201.580068331" watchObservedRunningTime="2026-11-04 00:39:45.689571058 +0000 UTC m=+4201.580632997"
Nov 04 00:39:45.772928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.772875    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" event={"ID":"fb4b017e-aceb-4f7d-8470-4af3187aea6b","Type":"ContainerStarted","Data":"f21c48f160725005ef9cbaf04056a025fe61441df61a716fbcd0fde00b6df881"}
Nov 04 00:39:45.773235 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.773213    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" event={"ID":"fb4b017e-aceb-4f7d-8470-4af3187aea6b","Type":"ContainerStarted","Data":"407c29cb988580f5404ed538c29a688bbfa662e8ea26ca7b66c368f57fc61e8e"}
Nov 04 00:39:45.805416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.802281    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podStartSLOduration=2.802260918 podStartE2EDuration="2.802260918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:45.79992697 +0000 UTC m=+4201.690988936" watchObservedRunningTime="2026-11-04 00:39:45.802260918 +0000 UTC m=+4201.693322853"
Nov 04 00:39:45.836327 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.836176    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" event={"ID":"c2ce493a-180f-47a6-a03e-74f3db88976f","Type":"ContainerStarted","Data":"d3978e06a33152e87da1739e8d81a067677652a65ebf0226629ce8544715ddf5"}
Nov 04 00:39:45.836673 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.836628    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" event={"ID":"c2ce493a-180f-47a6-a03e-74f3db88976f","Type":"ContainerStarted","Data":"a09e3eec6a17be093052c93a981955d2bfcf26c503e2af055e895ef458efd02f"}
Nov 04 00:39:45.867425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.858792    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:45.867425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.861766    6430 generic.go:356] "Generic (PLEG): container finished" podID="d23c108b-6f63-45bd-8c5c-2270a73646ff" containerID="61d6306d0475fbe2ad7567f5c604740499e13bef8c742272e9870b7d2fb2237c" exitCode=0
Nov 04 00:39:45.867425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.861850    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" event={"ID":"d23c108b-6f63-45bd-8c5c-2270a73646ff","Type":"ContainerDied","Data":"61d6306d0475fbe2ad7567f5c604740499e13bef8c742272e9870b7d2fb2237c"}
Nov 04 00:39:45.867425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.861921    6430 scope.go:122] "RemoveContainer" containerID="61d6306d0475fbe2ad7567f5c604740499e13bef8c742272e9870b7d2fb2237c"
Nov 04 00:39:45.901416 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.897971    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podStartSLOduration=2.897948985 podStartE2EDuration="2.897948985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:39:45.890321846 +0000 UTC m=+4201.781383782" watchObservedRunningTime="2026-11-04 00:39:45.897948985 +0000 UTC m=+4201.789010921"
Nov 04 00:39:45.911421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.907794    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\") pod \"d23c108b-6f63-45bd-8c5c-2270a73646ff\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") "
Nov 04 00:39:45.911421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.907908    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\") pod \"d23c108b-6f63-45bd-8c5c-2270a73646ff\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") "
Nov 04 00:39:45.911421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.907946    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\" (UniqueName: \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\") pod \"d23c108b-6f63-45bd-8c5c-2270a73646ff\" (UID: \"d23c108b-6f63-45bd-8c5c-2270a73646ff\") "
Nov 04 00:39:45.928609 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.928566    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs" pod "d23c108b-6f63-45bd-8c5c-2270a73646ff" (UID: "d23c108b-6f63-45bd-8c5c-2270a73646ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:45.975423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:45.957431    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert" pod "d23c108b-6f63-45bd-8c5c-2270a73646ff" (UID: "d23c108b-6f63-45bd-8c5c-2270a73646ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:39:46.033413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.026619    6430 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-webhook-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:46.033413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.026659    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d23c108b-6f63-45bd-8c5c-2270a73646ff-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:46.079871 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.071863    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9" pod "d23c108b-6f63-45bd-8c5c-2270a73646ff" (UID: "d23c108b-6f63-45bd-8c5c-2270a73646ff"). InnerVolumeSpecName "kube-api-access-t4zk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:39:46.177430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.174599    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4zk9\" (UniqueName: \"kubernetes.io/projected/d23c108b-6f63-45bd-8c5c-2270a73646ff-kube-api-access-t4zk9\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:39:46.321643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.321601    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:46.321989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.321955    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:46.329412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.325482    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:46.329684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.329650    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:46.375154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.370794    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:46.375154 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.370859    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:46.401481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.391253    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:46.461634 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.461600    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:46.466955 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.461997    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:46.467432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.467227    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:46.475573 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.429997    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:46.475972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.475852    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:46.896514 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.896458    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw" event={"ID":"d23c108b-6f63-45bd-8c5c-2270a73646ff","Type":"ContainerDied","Data":"8176c3dfebb9528af2121a00b72eb3870fc7cdfbb607d5482afe57b2fa893333"}
Nov 04 00:39:46.897500 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.897474    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc8d5ff6-6xqkw"
Nov 04 00:39:46.942413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.941740    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:46.947421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:46.947157    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cc8d5ff6-6xqkw"]
Nov 04 00:39:47.301411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.298747    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:47.301411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.298809    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:47.324413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.323779    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:47.324413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.323842    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:47.345417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.341576    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:47.345417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.341631    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:47.354412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.352492    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:47.354412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:47.352550    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:48.314020 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.313967    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:48.318528 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.318463    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:48.348241 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.348205    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:48.348571 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.348538    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:48.377422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.376583    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:48.377725 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.377689    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:48.377949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.377208    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:48.378197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.378175    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:48.406508 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:48.406471    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23c108b-6f63-45bd-8c5c-2270a73646ff" path="/var/lib/kubelet/pods/d23c108b-6f63-45bd-8c5c-2270a73646ff/volumes"
Nov 04 00:39:49.297413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.296869    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:49.297413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.296980    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:49.325412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.324778    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:49.325412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.324845    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:49.346414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.343777    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:49.346414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.343835    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:49.348159 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.348129    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:49.348458 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:49.348429    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:50.321296 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.298739    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:50.321296 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.298805    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:50.326780 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.326736    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:50.327346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.327312    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:50.340411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.339825    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:50.340411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.339896    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:50.362412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.348711    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:50.362412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:50.348766    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:51.296178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.296136    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:51.296421 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.296193    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:51.323789 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.323741    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:51.324000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.323797    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:51.337305 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.337264    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:51.337806 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.337318    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:51.348868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.348835    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:51.349042 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:51.348889    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:52.296401 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.296345    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 04 00:39:52.296589 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.296431    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 04 00:39:52.324311 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.324281    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body=
Nov 04 00:39:52.324506 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.324331    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused"
Nov 04 00:39:52.336719 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.336681    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Nov 04 00:39:52.336861 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.336732    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Nov 04 00:39:52.348422 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.348378    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Nov 04 00:39:52.348846 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:52.348443    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Nov 04 00:39:54.152028 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.151991    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:54.153960 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.153934    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:54.154471 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.154451    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:54.155172 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.155145    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:54.155537 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.155520    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4"
Nov 04 00:39:54.156992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.156948    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk"
Nov 04 00:39:54.190540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.189777    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:54.190540 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.190310    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:54.191001 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.190976    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg"
Nov 04 00:39:54.325326 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.325292    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:54.326195 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.326168    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:39:54.327339 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:39:54.327304    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5"
Nov 04 00:40:04.831107 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:40:04.831069    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:40:08.477718 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:40:08.477675    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:40:24.208738 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:40:24.208704    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:40:29.115414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:40:29.114947    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:40:32.520784 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:40:32.520746    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:40:34.911771 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:40:34.911682    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:40:47.769809 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:40:47.769776    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:41:04.830465 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:41:04.830419    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:41:14.911146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:41:14.911107    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:41:19.276182 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:41:19.276145    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:41:34.829117 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:41:34.829071    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:41:58.976445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:41:58.976342    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:42:04.831072 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:42:04.831034    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:42:28.109497 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:42:28.109461    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:42:34.828687 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:42:34.828647    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:42:40.266481 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:42:40.266447    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:43:04.830716 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:43:04.830674    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:43:05.365713 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.365672    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:43:05.366148 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.366129    6430 cpu_manager.go:414] "containerMap: removing container" podUID="d23c108b-6f63-45bd-8c5c-2270a73646ff" containerName="manager"
Nov 04 00:43:05.366249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.366151    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23c108b-6f63-45bd-8c5c-2270a73646ff" containerName="manager"
Nov 04 00:43:05.366314 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.366305    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23c108b-6f63-45bd-8c5c-2270a73646ff" containerName="manager"
Nov 04 00:43:05.369344 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.369322    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.375155 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.375129    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ca4fbe-openshift-default-dockercfg-ng5sr\""
Nov 04 00:43:05.383294 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.383265    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:43:05.465992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.465943    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.465992 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.465986    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466017    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466277 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466153    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466288    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2l8\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466427 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466335    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466559 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466449    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466559 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466489    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.466559 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.466514    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567696    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567737    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567863    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567897    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567924    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.567997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.567953    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568017    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2l8\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568050    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568117    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568122    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568242 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568196    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568283    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568346    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.568910 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.568885    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.570297 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.570278    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.570543 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.570526    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.581630 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.581602    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.581757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.581742    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2l8\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\") pod \"e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") " pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.683737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.683646    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:05.969815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.969778    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:43:05.975644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.974679    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:05.975644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.974815    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:05.975644 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:05.974903    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:06.154618 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:06.154572    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerStarted","Data":"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"}
Nov 04 00:43:06.154815 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:06.154640    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerStarted","Data":"dfe3d0727a156d71a4fe796190867bd8247473549fe30321d8b9e7af65143fe3"}
Nov 04 00:43:06.683803 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:06.683759    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:07.688949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:07.688919    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:07.717344 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:07.717292    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podStartSLOduration=2.7172743969999997 podStartE2EDuration="2.717274397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:43:06.182235737 +0000 UTC m=+4402.073297673" watchObservedRunningTime="2026-11-04 00:43:07.717274397 +0000 UTC m=+4403.608336326"
Nov 04 00:43:08.163372 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:08.163340    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:08.164313 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:08.164293    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:19.805141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.805098    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:43:19.810505 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.810472    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:19.817070 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.817031    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ca4fbe-serving-cert\""
Nov 04 00:43:19.824446 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.823415    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:43:19.849796 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.849752    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:43:19.853317 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.853289    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:19.861715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.861629    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:43:19.870766 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.870732    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ca4fbe\""
Nov 04 00:43:19.911487 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.911443    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9px\" (UniqueName: \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:19.911695 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.911616    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:19.911770 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.911721    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:19.911835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.911795    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:19.991840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.991799    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:43:19.995647 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:19.995617    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.013044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013010    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.013220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013073    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.013220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013094    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.013220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013126    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9px\" (UniqueName: \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.013220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013154    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljgb\" (UniqueName: \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.013220 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013201    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.013818 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.013797    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.015711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.015689    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.015771 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.015739    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.025184 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.025135    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0b757b52-dockercfg-6kqgf\""
Nov 04 00:43:20.025845 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.025821    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:43:20.032742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.032706    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9px\" (UniqueName: \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.114008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.113861    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rxq\" (UniqueName: \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.114008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.113945    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljgb\" (UniqueName: \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.114008 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.113996    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.114332 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.114226    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.115484 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.115460    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.128362 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.128331    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:20.128829 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.128806    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljgb\" (UniqueName: \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.177094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.177055    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:20.215043 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.214997    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.215247 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.215074    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rxq\" (UniqueName: \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.216400 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.216352    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.229157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.229118    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rxq\" (UniqueName: \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.308450 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.308404    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:20.419146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.419115    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:43:20.421423 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:43:20.421366    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf184d9c3_0898_45b8_96cb_b142ac4bd00f.slice/crio-3109c515532f2a05bc2da49cde9738af0b6a8b7b1055556c704492f902ca946d WatchSource:0}: Error finding container 3109c515532f2a05bc2da49cde9738af0b6a8b7b1055556c704492f902ca946d: Status 404 returned error can't find the container with id 3109c515532f2a05bc2da49cde9738af0b6a8b7b1055556c704492f902ca946d
Nov 04 00:43:20.472225 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.472195    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:43:20.474760 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:43:20.474734    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ac0d35_1073_4909_888e_5f5776d6cb21.slice/crio-8a12da7fb9c969417181b68d765306878a5c474abb4312bed1b14fab4f64013e WatchSource:0}: Error finding container 8a12da7fb9c969417181b68d765306878a5c474abb4312bed1b14fab4f64013e: Status 404 returned error can't find the container with id 8a12da7fb9c969417181b68d765306878a5c474abb4312bed1b14fab4f64013e
Nov 04 00:43:20.600840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:20.600810    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:43:20.603091 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:43:20.603059    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe1d0fd_0327_4ddd_8ab4_543410144431.slice/crio-c525045c05d3ee58073057b8d4808dfffb4ac3511d8ef94b47681115d95ab03b WatchSource:0}: Error finding container c525045c05d3ee58073057b8d4808dfffb4ac3511d8ef94b47681115d95ab03b: Status 404 returned error can't find the container with id c525045c05d3ee58073057b8d4808dfffb4ac3511d8ef94b47681115d95ab03b
Nov 04 00:43:21.229146 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.229043    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" event={"ID":"ebe1d0fd-0327-4ddd-8ab4-543410144431","Type":"ContainerStarted","Data":"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"}
Nov 04 00:43:21.229638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.229179    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" event={"ID":"ebe1d0fd-0327-4ddd-8ab4-543410144431","Type":"ContainerStarted","Data":"c525045c05d3ee58073057b8d4808dfffb4ac3511d8ef94b47681115d95ab03b"}
Nov 04 00:43:21.229638 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.229214    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:21.230943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.230915    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" event={"ID":"a1ac0d35-1073-4909-888e-5f5776d6cb21","Type":"ContainerStarted","Data":"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"}
Nov 04 00:43:21.230943 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.230966    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" event={"ID":"a1ac0d35-1073-4909-888e-5f5776d6cb21","Type":"ContainerStarted","Data":"8a12da7fb9c969417181b68d765306878a5c474abb4312bed1b14fab4f64013e"}
Nov 04 00:43:21.231181 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.230997    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:21.232601 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.232582    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" event={"ID":"f184d9c3-0898-45b8-96cb-b142ac4bd00f","Type":"ContainerStarted","Data":"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"}
Nov 04 00:43:21.232684 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.232616    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" event={"ID":"f184d9c3-0898-45b8-96cb-b142ac4bd00f","Type":"ContainerStarted","Data":"3109c515532f2a05bc2da49cde9738af0b6a8b7b1055556c704492f902ca946d"}
Nov 04 00:43:21.232758 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.232719    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:21.250760 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.250704    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" podStartSLOduration=2.250687552 podStartE2EDuration="2.250687552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:43:21.248677044 +0000 UTC m=+4417.139738962" watchObservedRunningTime="2026-11-04 00:43:21.250687552 +0000 UTC m=+4417.141749488"
Nov 04 00:43:21.271140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.271092    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" podStartSLOduration=1.724491379 podStartE2EDuration="2.271079593s" totalImagesPullingTime="546.588214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:43:21.270248233 +0000 UTC m=+4417.161310165" watchObservedRunningTime="2026-11-04 00:43:21.271079593 +0000 UTC m=+4417.162141529"
Nov 04 00:43:21.291257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:21.291198    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" podStartSLOduration=2.291180872 podStartE2EDuration="2.291180872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:43:21.289814837 +0000 UTC m=+4417.180876773" watchObservedRunningTime="2026-11-04 00:43:21.291180872 +0000 UTC m=+4417.182242807"
Nov 04 00:43:22.408769 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:22.408734    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:43:27.254643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:27.254604    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:43:32.240663 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.240632    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:43:32.241145 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.240704    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:43:32.301000 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.300962    6430 generic.go:356] "Generic (PLEG): container finished" podID="c86aea19-6616-4843-983d-40c4f8849dd5" containerID="5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53" exitCode=0
Nov 04 00:43:32.301175 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.301009    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerDied","Data":"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"}
Nov 04 00:43:32.301445 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.301429    6430 scope.go:122] "RemoveContainer" containerID="5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"
Nov 04 00:43:32.302852 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.302808    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:32.302979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.302901    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:32.302979 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:32.302952    6430 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 04 00:43:33.337606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:33.336847    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerStarted","Data":"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"}
Nov 04 00:43:33.684564 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:33.684470    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:33.685779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:33.685750    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 04 00:43:33.685915 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:33.685804    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 04 00:43:34.684880 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:34.684847    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 04 00:43:34.685267 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:34.684901    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 04 00:43:34.853057 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:43:34.853017    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:43:35.684335 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:35.684294    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 04 00:43:35.684567 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:35.684361    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 04 00:43:36.684668 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:36.684626    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 04 00:43:36.685081 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:36.684686    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 04 00:43:37.583324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:37.583281    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:43:37.684109 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:37.684068    6430 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 04 00:43:37.684315 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:37.684122    6430 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 04 00:43:38.163410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:38.163342    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:38.853206 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:38.853171    6430 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:38.854585 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:38.854560    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:43:54.495429 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:43:54.495377    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:44:04.837089 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:44:04.837048    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:44:17.071300 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:17.071259    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:44:22.846301 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:22.846258    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:44:22.854077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:22.854041    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:44:24.320699 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:24.320664    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:44:24.321249 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:24.320888    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" podUID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" containerName="maas-api" containerID="cri-o://d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a" gracePeriod=30
Nov 04 00:44:25.567848 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.567823    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:44:25.615866 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.615832    6430 generic.go:356] "Generic (PLEG): container finished" podID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" containerID="d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a" exitCode=0
Nov 04 00:44:25.615993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.615884    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"
Nov 04 00:44:25.615993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.615921    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" event={"ID":"f184d9c3-0898-45b8-96cb-b142ac4bd00f","Type":"ContainerDied","Data":"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"}
Nov 04 00:44:25.615993 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.615981    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs" event={"ID":"f184d9c3-0898-45b8-96cb-b142ac4bd00f","Type":"ContainerDied","Data":"3109c515532f2a05bc2da49cde9738af0b6a8b7b1055556c704492f902ca946d"}
Nov 04 00:44:25.616178 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.616003    6430 scope.go:122] "RemoveContainer" containerID="d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"
Nov 04 00:44:25.625257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.625241    6430 scope.go:122] "RemoveContainer" containerID="d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"
Nov 04 00:44:25.625642 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:44:25.625605    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a\": container with ID starting with d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a not found: ID does not exist" containerID="d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"
Nov 04 00:44:25.625810 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.625645    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a"} err="failed to get container status \"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a\": rpc error: code = NotFound desc = could not find container \"d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a\": container with ID starting with d9292dc51ee46d9049b3632b8fbf9bcc54e5c7f016125f4032e178318a5b5e7a not found: ID does not exist"
Nov 04 00:44:25.760808 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.760722    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\") pod \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") "
Nov 04 00:44:25.760986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.760905    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\") pod \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") "
Nov 04 00:44:25.760986 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.760964    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\" (UniqueName: \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\") pod \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") "
Nov 04 00:44:25.761118 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.760993    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\") pod \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\" (UID: \"f184d9c3-0898-45b8-96cb-b142ac4bd00f\") "
Nov 04 00:44:25.761431 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.761371    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca" pod "f184d9c3-0898-45b8-96cb-b142ac4bd00f" (UID: "f184d9c3-0898-45b8-96cb-b142ac4bd00f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:44:25.762909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.762886    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls" pod "f184d9c3-0898-45b8-96cb-b142ac4bd00f" (UID: "f184d9c3-0898-45b8-96cb-b142ac4bd00f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:44:25.763004 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.762935    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs" pod "f184d9c3-0898-45b8-96cb-b142ac4bd00f" (UID: "f184d9c3-0898-45b8-96cb-b142ac4bd00f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:44:25.763046 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.763022    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px" pod "f184d9c3-0898-45b8-96cb-b142ac4bd00f" (UID: "f184d9c3-0898-45b8-96cb-b142ac4bd00f"). InnerVolumeSpecName "kube-api-access-ph9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:44:25.862635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.862606    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:25.862635 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.862633    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9px\" (UniqueName: \"kubernetes.io/projected/f184d9c3-0898-45b8-96cb-b142ac4bd00f-kube-api-access-ph9px\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:25.862799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.862644    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9c3-0898-45b8-96cb-b142ac4bd00f-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:25.862799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.862654    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f184d9c3-0898-45b8-96cb-b142ac4bd00f-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:25.939648 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.939617    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:44:25.943568 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:25.943544    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-dtsvs"]
Nov 04 00:44:26.180237 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:26.180203    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:44:26.256140 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:26.256089    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" path="/var/lib/kubelet/pods/f184d9c3-0898-45b8-96cb-b142ac4bd00f/volumes"
Nov 04 00:44:27.847519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:27.847435    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" podUID="ebe1d0fd-0327-4ddd-8ab4-543410144431" containerName="payload-processing" containerID="cri-o://19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403" gracePeriod=25
Nov 04 00:44:27.854505 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:27.854457    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" podUID="a1ac0d35-1073-4909-888e-5f5776d6cb21" containerName="payload-pre-processing" containerID="cri-o://4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de" gracePeriod=25
Nov 04 00:44:28.098584 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.098515    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:44:28.138311 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.138279    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:44:28.182633 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.182584    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\" (UniqueName: \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\") pod \"a1ac0d35-1073-4909-888e-5f5776d6cb21\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") "
Nov 04 00:44:28.182633 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.182644    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\") pod \"ebe1d0fd-0327-4ddd-8ab4-543410144431\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") "
Nov 04 00:44:28.182921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.182689    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\") pod \"a1ac0d35-1073-4909-888e-5f5776d6cb21\" (UID: \"a1ac0d35-1073-4909-888e-5f5776d6cb21\") "
Nov 04 00:44:28.182921 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.182756    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\" (UniqueName: \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\") pod \"ebe1d0fd-0327-4ddd-8ab4-543410144431\" (UID: \"ebe1d0fd-0327-4ddd-8ab4-543410144431\") "
Nov 04 00:44:28.183144 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.183114    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume" pod "ebe1d0fd-0327-4ddd-8ab4-543410144431" (UID: "ebe1d0fd-0327-4ddd-8ab4-543410144431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:44:28.183292 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.183174    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume" pod "a1ac0d35-1073-4909-888e-5f5776d6cb21" (UID: "a1ac0d35-1073-4909-888e-5f5776d6cb21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:44:28.185240 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.185203    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb" pod "a1ac0d35-1073-4909-888e-5f5776d6cb21" (UID: "a1ac0d35-1073-4909-888e-5f5776d6cb21"). InnerVolumeSpecName "kube-api-access-kljgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:44:28.185367 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.185306    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq" pod "ebe1d0fd-0327-4ddd-8ab4-543410144431" (UID: "ebe1d0fd-0327-4ddd-8ab4-543410144431"). InnerVolumeSpecName "kube-api-access-79rxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:44:28.284324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.284287    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kljgb\" (UniqueName: \"kubernetes.io/projected/a1ac0d35-1073-4909-888e-5f5776d6cb21-kube-api-access-kljgb\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:28.284324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.284319    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe1d0fd-0327-4ddd-8ab4-543410144431-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:28.284324 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.284328    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ac0d35-1073-4909-888e-5f5776d6cb21-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:28.284584 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.284337    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rxq\" (UniqueName: \"kubernetes.io/projected/ebe1d0fd-0327-4ddd-8ab4-543410144431-kube-api-access-79rxq\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:44:28.632425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.632366    6430 generic.go:356] "Generic (PLEG): container finished" podID="ebe1d0fd-0327-4ddd-8ab4-543410144431" containerID="19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403" exitCode=0
Nov 04 00:44:28.632617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.632427    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"
Nov 04 00:44:28.632617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.632458    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" event={"ID":"ebe1d0fd-0327-4ddd-8ab4-543410144431","Type":"ContainerDied","Data":"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"}
Nov 04 00:44:28.632617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.632507    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq" event={"ID":"ebe1d0fd-0327-4ddd-8ab4-543410144431","Type":"ContainerDied","Data":"c525045c05d3ee58073057b8d4808dfffb4ac3511d8ef94b47681115d95ab03b"}
Nov 04 00:44:28.632617 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.632536    6430 scope.go:122] "RemoveContainer" containerID="19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"
Nov 04 00:44:28.633908 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.633884    6430 generic.go:356] "Generic (PLEG): container finished" podID="a1ac0d35-1073-4909-888e-5f5776d6cb21" containerID="4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de" exitCode=0
Nov 04 00:44:28.634044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.633916    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" event={"ID":"a1ac0d35-1073-4909-888e-5f5776d6cb21","Type":"ContainerDied","Data":"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"}
Nov 04 00:44:28.634044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.633938    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"
Nov 04 00:44:28.634044 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.633968    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk" event={"ID":"a1ac0d35-1073-4909-888e-5f5776d6cb21","Type":"ContainerDied","Data":"8a12da7fb9c969417181b68d765306878a5c474abb4312bed1b14fab4f64013e"}
Nov 04 00:44:28.650606 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.650582    6430 scope.go:122] "RemoveContainer" containerID="19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"
Nov 04 00:44:28.650881 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:44:28.650858    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403\": container with ID starting with 19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403 not found: ID does not exist" containerID="19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"
Nov 04 00:44:28.650963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.650903    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403"} err="failed to get container status \"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403\": rpc error: code = NotFound desc = could not find container \"19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403\": container with ID starting with 19586800a1f16399ed89bb526fb1b3698f5c4accc33cc91358c52602e08a8403 not found: ID does not exist"
Nov 04 00:44:28.650963 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.650920    6430 scope.go:122] "RemoveContainer" containerID="4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"
Nov 04 00:44:28.658604 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.658587    6430 scope.go:122] "RemoveContainer" containerID="4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"
Nov 04 00:44:28.658870 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:44:28.658852    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de\": container with ID starting with 4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de not found: ID does not exist" containerID="4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"
Nov 04 00:44:28.658919 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.658877    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de"} err="failed to get container status \"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de\": rpc error: code = NotFound desc = could not find container \"4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de\": container with ID starting with 4f7f3d7819a9df04c3d20232fe9883e70d3dde83f2483707b0e6a084818961de not found: ID does not exist"
Nov 04 00:44:28.662300 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.662280    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:44:28.668468 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.668444    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff857n7bk"]
Nov 04 00:44:28.684971 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.684944    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:44:28.687744 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:28.687723    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc4nqqq"]
Nov 04 00:44:30.229678 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:30.229647    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ac0d35-1073-4909-888e-5f5776d6cb21" path="/var/lib/kubelet/pods/a1ac0d35-1073-4909-888e-5f5776d6cb21/volumes"
Nov 04 00:44:30.230059 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:30.230023    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe1d0fd-0327-4ddd-8ab4-543410144431" path="/var/lib/kubelet/pods/ebe1d0fd-0327-4ddd-8ab4-543410144431/volumes"
Nov 04 00:44:34.830175 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:44:34.830134    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:44:58.703613 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:44:58.703574    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:45:01.908062 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:01.908027    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:45:04.648098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648061    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"]
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648472    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ebe1d0fd-0327-4ddd-8ab4-543410144431" containerName="payload-processing"
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648486    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe1d0fd-0327-4ddd-8ab4-543410144431" containerName="payload-processing"
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648509    6430 cpu_manager.go:414] "containerMap: removing container" podUID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" containerName="maas-api"
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648514    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" containerName="maas-api"
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648533    6430 cpu_manager.go:414] "containerMap: removing container" podUID="a1ac0d35-1073-4909-888e-5f5776d6cb21" containerName="payload-pre-processing"
Nov 04 00:45:04.648535 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648538    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ac0d35-1073-4909-888e-5f5776d6cb21" containerName="payload-pre-processing"
Nov 04 00:45:04.648776 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648600    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe1d0fd-0327-4ddd-8ab4-543410144431" containerName="payload-processing"
Nov 04 00:45:04.648776 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648612    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="f184d9c3-0898-45b8-96cb-b142ac4bd00f" containerName="maas-api"
Nov 04 00:45:04.648776 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.648620    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ac0d35-1073-4909-888e-5f5776d6cb21" containerName="payload-pre-processing"
Nov 04 00:45:04.651763 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.651742    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.661246 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.661219    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\""
Nov 04 00:45:04.662040 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.662011    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 04 00:45:04.662989 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.662962    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"]
Nov 04 00:45:04.831261 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:45:04.831224    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:45:04.844835 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.844805    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6624\" (UniqueName: \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.844997 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.844941    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.845041 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.845014    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.945643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.945550    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.945643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.945606    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.945643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.945637    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6624\" (UniqueName: \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.946258 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.946223    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.948036 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.948005    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.958976 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.958947    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6624\" (UniqueName: \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:04.966769 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:04.966748    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:05.239245 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:05.239216    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"]
Nov 04 00:45:05.810080 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:05.810040    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8" event={"ID":"ca8f160b-e237-4ea8-ba15-073d523aee42","Type":"ContainerStarted","Data":"d07f56ba4706ea75cf0f7a862e0d84ceed26da8e80f876d0aad3aae2e025374d"}
Nov 04 00:45:05.810500 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:05.810106    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8" event={"ID":"ca8f160b-e237-4ea8-ba15-073d523aee42","Type":"ContainerStarted","Data":"5a0cc390f023a88ea08aac4e3cda200518d2621162b7b3fb3af6aed13a884bc2"}
Nov 04 00:45:05.832149 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:05.832097    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8" podStartSLOduration=1.832083221 podStartE2EDuration="1.832083221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:45:05.829544343 +0000 UTC m=+4521.720606305" watchObservedRunningTime="2026-11-04 00:45:05.832083221 +0000 UTC m=+4521.723145156"
Nov 04 00:45:06.815881 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:06.815838    6430 generic.go:356] "Generic (PLEG): container finished" podID="ca8f160b-e237-4ea8-ba15-073d523aee42" containerID="d07f56ba4706ea75cf0f7a862e0d84ceed26da8e80f876d0aad3aae2e025374d" exitCode=0
Nov 04 00:45:06.816264 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:06.815907    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8" event={"ID":"ca8f160b-e237-4ea8-ba15-073d523aee42","Type":"ContainerDied","Data":"d07f56ba4706ea75cf0f7a862e0d84ceed26da8e80f876d0aad3aae2e025374d"}
Nov 04 00:45:07.956930 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.956906    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:07.972533 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.972511    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\" (UniqueName: \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\") pod \"ca8f160b-e237-4ea8-ba15-073d523aee42\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") "
Nov 04 00:45:07.972641 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.972606    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\") pod \"ca8f160b-e237-4ea8-ba15-073d523aee42\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") "
Nov 04 00:45:07.972715 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.972674    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\") pod \"ca8f160b-e237-4ea8-ba15-073d523aee42\" (UID: \"ca8f160b-e237-4ea8-ba15-073d523aee42\") "
Nov 04 00:45:07.972932 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.972910    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca" pod "ca8f160b-e237-4ea8-ba15-073d523aee42" (UID: "ca8f160b-e237-4ea8-ba15-073d523aee42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:45:07.975629 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.975601    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca" pod "ca8f160b-e237-4ea8-ba15-073d523aee42" (UID: "ca8f160b-e237-4ea8-ba15-073d523aee42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:45:07.983253 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:07.983193    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624" pod "ca8f160b-e237-4ea8-ba15-073d523aee42" (UID: "ca8f160b-e237-4ea8-ba15-073d523aee42"). InnerVolumeSpecName "kube-api-access-t6624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:45:08.073552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.073465    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:08.073552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.073498    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8f160b-e237-4ea8-ba15-073d523aee42-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:08.073552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.073511    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6624\" (UniqueName: \"kubernetes.io/projected/ca8f160b-e237-4ea8-ba15-073d523aee42-kube-api-access-t6624\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:08.825739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.825699    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8" event={"ID":"ca8f160b-e237-4ea8-ba15-073d523aee42","Type":"ContainerDied","Data":"5a0cc390f023a88ea08aac4e3cda200518d2621162b7b3fb3af6aed13a884bc2"}
Nov 04 00:45:08.825739 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.825723    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"
Nov 04 00:45:08.825949 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:08.825748    6430 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a0cc390f023a88ea08aac4e3cda200518d2621162b7b3fb3af6aed13a884bc2"
Nov 04 00:45:15.149705 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:15.149666    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"]
Nov 04 00:45:15.152470 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:15.152443    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca4fbe-txmg8"]
Nov 04 00:45:15.519586 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:15.519505    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:45:15.519799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:15.519771    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy" containerID="cri-o://3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d" gracePeriod=30
Nov 04 00:45:16.229868 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:16.229834    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8f160b-e237-4ea8-ba15-073d523aee42" path="/var/lib/kubelet/pods/ca8f160b-e237-4ea8-ba15-073d523aee42/volumes"
Nov 04 00:45:17.050224 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:17.050188    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:45:20.771465 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.771440    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:45:20.813012 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.812984    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813027    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813082    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813152    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813186 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813181    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813229    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813253    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813296    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813451 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813363    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\") pod \"c86aea19-6616-4843-983d-40c4f8849dd5\" (UID: \"c86aea19-6616-4843-983d-40c4f8849dd5\") "
Nov 04 00:45:20.813670 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813519    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:45:20.813757 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813732    6430 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-credential-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.814141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813841    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:45:20.814141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.813860    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:45:20.814141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.814018    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:45:20.814141 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.814104    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:45:20.815532 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.815507    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 04 00:45:20.815696 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.815668    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:45:20.815799 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.815774    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "kube-api-access-qd2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:45:20.816353 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.816330    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy" pod "c86aea19-6616-4843-983d-40c4f8849dd5" (UID: "c86aea19-6616-4843-983d-40c4f8849dd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 04 00:45:20.885489 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.885446    6430 generic.go:356] "Generic (PLEG): container finished" podID="c86aea19-6616-4843-983d-40c4f8849dd5" containerID="3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d" exitCode=0
Nov 04 00:45:20.885669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.885516    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"
Nov 04 00:45:20.885669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.885513    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerDied","Data":"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"}
Nov 04 00:45:20.885669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.885650    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6" event={"ID":"c86aea19-6616-4843-983d-40c4f8849dd5","Type":"ContainerDied","Data":"dfe3d0727a156d71a4fe796190867bd8247473549fe30321d8b9e7af65143fe3"}
Nov 04 00:45:20.885812 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.885682    6430 scope.go:122] "RemoveContainer" containerID="3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"
Nov 04 00:45:20.895749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.895733    6430 scope.go:122] "RemoveContainer" containerID="5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"
Nov 04 00:45:20.904170 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.904150    6430 scope.go:122] "RemoveContainer" containerID="3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"
Nov 04 00:45:20.904436 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:45:20.904416    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d\": container with ID starting with 3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d not found: ID does not exist" containerID="3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"
Nov 04 00:45:20.904491 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.904445    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d"} err="failed to get container status \"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d\": rpc error: code = NotFound desc = could not find container \"3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d\": container with ID starting with 3324f88bcfeffcc98fe6d8fad33b499ae27dfd525b8755b462da89483531d37d not found: ID does not exist"
Nov 04 00:45:20.904491 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.904462    6430 scope.go:122] "RemoveContainer" containerID="5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"
Nov 04 00:45:20.904713 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:45:20.904693    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53\": container with ID starting with 5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53 not found: ID does not exist" containerID="5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"
Nov 04 00:45:20.904763 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.904719    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53"} err="failed to get container status \"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53\": rpc error: code = NotFound desc = could not find container \"5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53\": container with ID starting with 5feb2765e959fdfeb12f7b8866de89053a9f3dea77f8b4afb780a6114b236e53 not found: ID does not exist"
Nov 04 00:45:20.911197 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.911172    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:45:20.914474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914447    6430 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-data\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914474 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914473    6430 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-socket\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914484    6430 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-istio-token\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914493    6430 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-workload-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914502    6430 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86aea19-6616-4843-983d-40c4f8849dd5-istiod-ca-cert\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914511    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2l8\" (UniqueName: \"kubernetes.io/projected/c86aea19-6616-4843-983d-40c4f8849dd5-kube-api-access-qd2l8\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914521    6430 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86aea19-6616-4843-983d-40c4f8849dd5-istio-envoy\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.914645 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.914530    6430 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86aea19-6616-4843-983d-40c4f8849dd5-istio-podinfo\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:45:20.915346 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:20.915324    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca4fbe-openshift-default-67b69f86ccfxlx6"]
Nov 04 00:45:22.230603 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:22.230570    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" path="/var/lib/kubelet/pods/c86aea19-6616-4843-983d-40c4f8849dd5/volumes"
Nov 04 00:45:22.491518 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:22.491418    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:45:25.201548 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.201499    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:25.202196 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.202164    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:25.202756 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.202733    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:25.202928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.202907    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:25.247417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.245506    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:25.247417 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.245590    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:25.372972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.372572    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:45:25.372972 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:25.372666    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:45:34.831038 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:45:34.831000    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:45:38.883811 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:38.883769    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:45:54.109675 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:45:54.109633    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:46:04.829888 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:46:04.829846    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:46:05.871169 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:46:05.871125    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:46:12.020508 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:46:12.020449    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:46:28.770877 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:46:28.770844    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:46:33.417525 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:46:33.417461    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:46:34.829187 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:46:34.829151    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:46:46.392824 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:46:46.392788    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:47:04.829623 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:47:04.829586    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:47:34.829245 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:47:34.829200    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:48:04.278157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:48:04.278079    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:48:04.844092 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:48:04.844027    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:48:34.829381 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:48:34.829343    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:49:04.829486 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:49:04.829441    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:49:34.829184 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:49:34.829099    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:49:40.564619 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:49:40.564581    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:49:44.794413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:49:44.793505    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:50:00.789843 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:50:00.789796    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:50:05.005411 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:50:05.004594    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:50:05.771711 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:50:05.771671    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:50:17.672163 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:50:17.672127    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:50:30.615789 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:50:30.615721    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:50:34.828887 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:50:34.828845    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:50:55.290586 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:50:55.290514    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:04.829551 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:51:04.829512    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:51:05.565779 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:05.565749    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:10.203411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.202463    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:51:10.203411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.202531    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:51:10.203411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.202948    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:51:10.203411 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.202983    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:51:10.245410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.242578    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:51:10.245410 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.242677    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:51:10.436774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:10.436726    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:26.311596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:26.311559    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:30.306519 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:30.306476    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:33.705882 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:33.705840    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:34.829675 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:51:34.829637    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:51:44.619643 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:44.619603    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:51:55.219080 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:55.219049    6430 scope.go:122] "RemoveContainer" containerID="d07f56ba4706ea75cf0f7a862e0d84ceed26da8e80f876d0aad3aae2e025374d"
Nov 04 00:51:58.510591 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:51:58.510548    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:02.593412 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:02.589381    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:04.829836 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:52:04.829797    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:52:05.878659 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:05.878621    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:23.880463 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:23.880371    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:34.610288 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:34.610253    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:34.831723 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:52:34.831687    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:52:48.974493 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:48.974447    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:52.496991 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:52.496951    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:52:56.589710 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:52:56.589672    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:53:04.830454 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:53:04.830413    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:53:21.572459 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:53:21.572425    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:53:31.791977 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:53:31.791948    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:53:34.829195 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:53:34.829157    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:54:04.829518 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:54:04.829431    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:54:18.798094 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:18.798059    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:54:23.467168 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:23.467133    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:54:34.829504 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:54:34.829464    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:54:37.960740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.960703    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961100    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961110    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961137    6430 cpu_manager.go:414] "containerMap: removing container" podUID="ca8f160b-e237-4ea8-ba15-073d523aee42" containerName="revoke-keys"
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961142    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8f160b-e237-4ea8-ba15-073d523aee42" containerName="revoke-keys"
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961203    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:54:37.961447 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.961217    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8f160b-e237-4ea8-ba15-073d523aee42" containerName="revoke-keys"
Nov 04 00:54:37.964477 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.964454    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:37.978864 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.978837    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-15b71d4f-dockercfg-xkqqx\""
Nov 04 00:54:37.980454 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:37.980431    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:54:38.011610 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.011582    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7t8\" (UniqueName: \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.011740 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.011628    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.012782 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.012767    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b34223\""
Nov 04 00:54:38.112576 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.112542    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7t8\" (UniqueName: \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.112773 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.112589    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.113325 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.113302    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.125669 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.125636    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7t8\" (UniqueName: \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.277823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.277736    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:38.558928 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:38.553726    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:54:38.560701 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:54:38.560671    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f80c350_9ebd_4b59_913e_125341c793fb.slice/crio-53d37bc158b54a2abf28ab7fe2d14d79c6b05d127ec790a85fcbf06e9d2af396 WatchSource:0}: Error finding container 53d37bc158b54a2abf28ab7fe2d14d79c6b05d127ec790a85fcbf06e9d2af396: Status 404 returned error can't find the container with id 53d37bc158b54a2abf28ab7fe2d14d79c6b05d127ec790a85fcbf06e9d2af396
Nov 04 00:54:39.311732 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:39.311697    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" event={"ID":"8f80c350-9ebd-4b59-913e-125341c793fb","Type":"ContainerStarted","Data":"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"}
Nov 04 00:54:39.312152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:39.311746    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" event={"ID":"8f80c350-9ebd-4b59-913e-125341c793fb","Type":"ContainerStarted","Data":"53d37bc158b54a2abf28ab7fe2d14d79c6b05d127ec790a85fcbf06e9d2af396"}
Nov 04 00:54:39.312152 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:39.311826    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:54:39.332526 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:39.332481    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" podStartSLOduration=2.332469127 podStartE2EDuration="2.332469127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:54:39.33141134 +0000 UTC m=+5095.222473275" watchObservedRunningTime="2026-11-04 00:54:39.332469127 +0000 UTC m=+5095.223531057"
Nov 04 00:54:40.678414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:40.678356    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:54:41.509938 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:41.509886    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" podUID="b6a44b6090d9fc279a15fb8a9a158ae7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:54:50.319122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:54:50.319092    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:55:04.830002 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:55:04.829962    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:55:06.196077 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:55:06.196039    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:55:32.725680 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:55:32.725601    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:55:34.829111 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:55:34.829074    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:55:55.879179 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:55:55.879142    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:56:01.517356 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:01.517312    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:56:04.712784 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:04.712746    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:56:04.829717 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:56:04.829685    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:56:06.517948 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:06.517885    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing" containerID="cri-o://4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5" gracePeriod=25
Nov 04 00:56:10.317727 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:10.317677    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:56:20.317322 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:20.317278    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 04 00:56:26.757336 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.757310    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:56:26.781909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.781838    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\") pod \"8f80c350-9ebd-4b59-913e-125341c793fb\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") "
Nov 04 00:56:26.781909 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.781901    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\" (UniqueName: \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\") pod \"8f80c350-9ebd-4b59-913e-125341c793fb\" (UID: \"8f80c350-9ebd-4b59-913e-125341c793fb\") "
Nov 04 00:56:26.782425 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.782367    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume" pod "8f80c350-9ebd-4b59-913e-125341c793fb" (UID: "8f80c350-9ebd-4b59-913e-125341c793fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:56:26.784304 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.784277    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8" pod "8f80c350-9ebd-4b59-913e-125341c793fb" (UID: "8f80c350-9ebd-4b59-913e-125341c793fb"). InnerVolumeSpecName "kube-api-access-rc7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:56:26.814945 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.814916    6430 generic.go:356] "Generic (PLEG): container finished" podID="8f80c350-9ebd-4b59-913e-125341c793fb" containerID="4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5" exitCode=0
Nov 04 00:56:26.815087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.814971    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"
Nov 04 00:56:26.815087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.815001    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" event={"ID":"8f80c350-9ebd-4b59-913e-125341c793fb","Type":"ContainerDied","Data":"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"}
Nov 04 00:56:26.815087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.815055    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs" event={"ID":"8f80c350-9ebd-4b59-913e-125341c793fb","Type":"ContainerDied","Data":"53d37bc158b54a2abf28ab7fe2d14d79c6b05d127ec790a85fcbf06e9d2af396"}
Nov 04 00:56:26.815087 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.815075    6430 scope.go:122] "RemoveContainer" containerID="4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"
Nov 04 00:56:26.823717 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.823697    6430 scope.go:122] "RemoveContainer" containerID="4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"
Nov 04 00:56:26.823960 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:56:26.823938    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5\": container with ID starting with 4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5 not found: ID does not exist" containerID="4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"
Nov 04 00:56:26.824021 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.823972    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5"} err="failed to get container status \"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5\": rpc error: code = NotFound desc = could not find container \"4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5\": container with ID starting with 4159df70728a4ba7b001cbc1e28410fd929e7f7ed87674168c162f79576144a5 not found: ID does not exist"
Nov 04 00:56:26.840139 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.840108    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:56:26.844679 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.844654    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfkvpcs"]
Nov 04 00:56:26.882840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.882814    6430 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f80c350-9ebd-4b59-913e-125341c793fb-plugins-config-volume\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:26.882840 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:26.882837    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7t8\" (UniqueName: \"kubernetes.io/projected/8f80c350-9ebd-4b59-913e-125341c793fb-kube-api-access-rc7t8\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:56:28.229889 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:28.229856    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" path="/var/lib/kubelet/pods/8f80c350-9ebd-4b59-913e-125341c793fb/volumes"
Nov 04 00:56:34.830207 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:56:34.830166    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:56:36.574737 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:36.574700    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:56:39.721283 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:39.721244    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:56:56.978512 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:56:56.978471    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:57:04.829928 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:57:04.829889    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:57:08.302135 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:08.302105    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:57:10.201414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.198528    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:57:10.201414 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.198602    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:57:10.217557 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.217523    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:57:10.217817 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.217794    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:57:10.245490 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.245456    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:57:10.245749 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.245720    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:57:10.377545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.377485    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:57:10.377892 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:57:10.377842    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:57:34.829803 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:57:34.829768    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:58:04.830711 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:58:04.830672    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:58:31.402002 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:58:31.401968    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:58:34.830143 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:58:34.830099    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:58:55.501716 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:58:55.501683    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:59:00.151423 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:00.147853    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:59:03.176166 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:03.176128    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:59:04.829925 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:59:04.829892    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:59:21.206552 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.206516    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"]
Nov 04 00:59:21.207053 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207032    6430 cpu_manager.go:414] "containerMap: removing container" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing"
Nov 04 00:59:21.207122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207054    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing"
Nov 04 00:59:21.207122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207072    6430 cpu_manager.go:414] "containerMap: removing container" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:59:21.207122 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207078    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:59:21.207280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207158    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f80c350-9ebd-4b59-913e-125341c793fb" containerName="payload-pre-processing"
Nov 04 00:59:21.207280 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.207169    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86aea19-6616-4843-983d-40c4f8849dd5" containerName="istio-proxy"
Nov 04 00:59:21.210430 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.210411    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.218359 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.218318    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"]
Nov 04 00:59:21.277377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.277350    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-maas-api-tls\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.277377 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.277402    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-metrics-certs\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.277660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.277547    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f807f-4de5-471d-be61-21aca227f00b-openshift-service-ca\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.277660 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.277645    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt62q\" (UniqueName: \"kubernetes.io/projected/8f5f807f-4de5-471d-be61-21aca227f00b-kube-api-access-gt62q\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.378343 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.378313    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-maas-api-tls\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.378560 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.378355    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-metrics-certs\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.378560 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.378421    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f807f-4de5-471d-be61-21aca227f00b-openshift-service-ca\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.378560 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.378486    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt62q\" (UniqueName: \"kubernetes.io/projected/8f5f807f-4de5-471d-be61-21aca227f00b-kube-api-access-gt62q\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.379111 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.379089    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5f807f-4de5-471d-be61-21aca227f00b-openshift-service-ca\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.380903 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.380877    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-metrics-certs\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.380984 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.380969    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5f807f-4de5-471d-be61-21aca227f00b-maas-api-tls\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.391257 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.391228    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt62q\" (UniqueName: \"kubernetes.io/projected/8f5f807f-4de5-471d-be61-21aca227f00b-kube-api-access-gt62q\") pod \"maas-api-6fcfd65cff-9bqkw\" (UID: \"8f5f807f-4de5-471d-be61-21aca227f00b\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.524742 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.524673    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:21.802908 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.802883    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"]
Nov 04 00:59:21.804966 ip-10-0-143-134 kubenswrapper[6430]: W1104 00:59:21.804931    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f807f_4de5_471d_be61_21aca227f00b.slice/crio-b34aa4745c45ae4f3791cc694db2dc11090aac96c9ed90fe83c97af5b7c2d847 WatchSource:0}: Error finding container b34aa4745c45ae4f3791cc694db2dc11090aac96c9ed90fe83c97af5b7c2d847: Status 404 returned error can't find the container with id b34aa4745c45ae4f3791cc694db2dc11090aac96c9ed90fe83c97af5b7c2d847
Nov 04 00:59:21.806275 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.806256    6430 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 04 00:59:21.987834 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:21.987795    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 00:59:22.694053 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:22.694004    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw" event={"ID":"8f5f807f-4de5-471d-be61-21aca227f00b","Type":"ContainerStarted","Data":"fe528d698ebc21a1d53c0ac22df8b1372f944c20c1036155670edbbe84856d56"}
Nov 04 00:59:22.694596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:22.694083    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw" event={"ID":"8f5f807f-4de5-471d-be61-21aca227f00b","Type":"ContainerStarted","Data":"b34aa4745c45ae4f3791cc694db2dc11090aac96c9ed90fe83c97af5b7c2d847"}
Nov 04 00:59:22.694596 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:22.694126    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:22.718082 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:22.718020    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw" podStartSLOduration=1.279413309 podStartE2EDuration="1.718002195s" totalImagesPullingTime="438.588886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:59:22.717197941 +0000 UTC m=+5378.608259874" watchObservedRunningTime="2026-11-04 00:59:22.718002195 +0000 UTC m=+5378.609064130"
Nov 04 00:59:25.156107 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:25.156056    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:59:25.156671 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:25.156626    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:59:25.176575 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:25.176547    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 00:59:25.176787 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:25.176762    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 00:59:28.772413 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:28.770556    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-9bqkw"
Nov 04 00:59:28.845486 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:28.845433    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 04 00:59:28.846476 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:28.846219    6430 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" podUID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" containerName="maas-api" containerID="cri-o://72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0" gracePeriod=30
Nov 04 00:59:30.152276 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.152250    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 04 00:59:30.288545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.288456    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\") pod \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") "
Nov 04 00:59:30.288545 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.288510    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\") pod \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") "
Nov 04 00:59:30.288823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.288592    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\" (UniqueName: \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\") pod \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") "
Nov 04 00:59:30.288823 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.288636    6430 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\") pod \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\" (UID: \"59db20c6-b9de-4de9-b16d-48a648fc7e8b\") "
Nov 04 00:59:30.289100 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.289070    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca" pod "59db20c6-b9de-4de9-b16d-48a648fc7e8b" (UID: "59db20c6-b9de-4de9-b16d-48a648fc7e8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 04 00:59:30.290657 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.290628    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls" pod "59db20c6-b9de-4de9-b16d-48a648fc7e8b" (UID: "59db20c6-b9de-4de9-b16d-48a648fc7e8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:59:30.290774 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.290758    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs" pod "59db20c6-b9de-4de9-b16d-48a648fc7e8b" (UID: "59db20c6-b9de-4de9-b16d-48a648fc7e8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 04 00:59:30.290853 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.290833    6430 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv" pod "59db20c6-b9de-4de9-b16d-48a648fc7e8b" (UID: "59db20c6-b9de-4de9-b16d-48a648fc7e8b"). InnerVolumeSpecName "kube-api-access-dh6mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 04 00:59:30.390432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.390383    6430 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6mv\" (UniqueName: \"kubernetes.io/projected/59db20c6-b9de-4de9-b16d-48a648fc7e8b-kube-api-access-dh6mv\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:30.390432 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.390431    6430 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59db20c6-b9de-4de9-b16d-48a648fc7e8b-openshift-service-ca\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:30.390592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.390441    6430 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-maas-api-tls\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:30.390592 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.390450    6430 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59db20c6-b9de-4de9-b16d-48a648fc7e8b-metrics-certs\") on node \"ip-10-0-143-134.us-east-2.compute.internal\" DevicePath \"\""
Nov 04 00:59:30.815311 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.815270    6430 generic.go:356] "Generic (PLEG): container finished" podID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" containerID="72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0" exitCode=0
Nov 04 00:59:30.815538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.815318    6430 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm"
Nov 04 00:59:30.815538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.815357    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" event={"ID":"59db20c6-b9de-4de9-b16d-48a648fc7e8b","Type":"ContainerDied","Data":"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"}
Nov 04 00:59:30.815538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.815457    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-g9klm" event={"ID":"59db20c6-b9de-4de9-b16d-48a648fc7e8b","Type":"ContainerDied","Data":"bce86a196216d181557a436c38041384e95205022283967c502b690590992d3f"}
Nov 04 00:59:30.815538 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.815484    6430 scope.go:122] "RemoveContainer" containerID="72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"
Nov 04 00:59:30.825355 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.825336    6430 scope.go:122] "RemoveContainer" containerID="72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"
Nov 04 00:59:30.825639 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:59:30.825610    6430 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0\": container with ID starting with 72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0 not found: ID does not exist" containerID="72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"
Nov 04 00:59:30.825721 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.825641    6430 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0"} err="failed to get container status \"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0\": rpc error: code = NotFound desc = could not find container \"72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0\": container with ID starting with 72b250e68b17a3745901c4e90189f3fb7fd8524ec10e1a120f73b0dc8250edf0 not found: ID does not exist"
Nov 04 00:59:30.839098 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.839072    6430 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 04 00:59:30.843115 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:30.843092    6430 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-g9klm"]
Nov 04 00:59:32.229542 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:32.229507    6430 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" path="/var/lib/kubelet/pods/59db20c6-b9de-4de9-b16d-48a648fc7e8b/volumes"
Nov 04 00:59:34.830110 ip-10-0-143-134 kubenswrapper[6430]: E1104 00:59:34.830068    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 00:59:51.970157 ip-10-0-143-134 kubenswrapper[6430]: I1104 00:59:51.970120    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:00:04.377045 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:00:04.377009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:00:04.846142 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:00:04.846078    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:00:21.965465 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:00:21.965422    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:00:34.004446 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:00:34.004410    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:00:34.844101 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:00:34.844056    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:00:38.009048 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:00:38.009009    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:00:59.608377 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:00:59.608333    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:04.830762 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:01:04.830717    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:01:09.702636 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:09.702596    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:15.196070 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:15.195681    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:21.308909 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:21.308870    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:34.830277 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:01:34.830187    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:01:37.864874 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:37.864837    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:40.212319 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.212278    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:01:40.212957 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.212925    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:01:40.216497 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.216466    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:01:40.216713 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.216692    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:01:40.235026 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.234633    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:01:40.235026 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.234705    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:01:40.373610 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.373548    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:01:40.373826 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:40.373639    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:01:47.966695 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:47.966661    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:01:52.646411 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:01:52.645499    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:02:04.831140 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:02:04.831093    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:02:16.289412 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:16.288167    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:02:32.313774 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:32.313727    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:02:35.024710 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:02:35.024588    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:02:55.201180 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.200507    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-9t5h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:02:55.201180 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.200586    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-9t5h4" podUID="eb9de502-2a33-42ea-996d-f96b86c57a3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:02:55.202942 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.202846    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-x42sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:02:55.202942 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.202892    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-x42sk" podUID="fb4b017e-aceb-4f7d-8470-4af3187aea6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:02:55.243793 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.243754    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-tcztg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:02:55.244090 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.244057    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tcztg" podUID="c2ce493a-180f-47a6-a03e-74f3db88976f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:02:55.380413 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.375548    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:02:55.380413 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:02:55.375621    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:03:04.885622 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:03:04.885582    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:03:25.382415 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:03:25.376911    6430 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-sjtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 04 01:03:25.382415 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:03:25.376996    6430 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-sjtz5" podUID="8035f969-cba1-40f0-904d-ee65b610b96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:03:34.951376 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:03:34.951319    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:04:04.844671 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:04:04.844622    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:04:24.374151 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:04:24.374052    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jt564"]
Nov 04 01:04:34.830352 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:04:34.830309    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:05:03.107957 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:03.107922    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:05:04.834603 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:05:04.834559    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:05:14.750944 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:14.750904    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fcfd65cff-9bqkw_8f5f807f-4de5-471d-be61-21aca227f00b/maas-api/0.log"
Nov 04 01:05:30.498137 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:30.498084    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 01:05:34.830247 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:05:34.830207    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:05:41.133724 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:41.133694    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 01:05:41.331048 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:41.331016    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"
Nov 04 01:05:41.928735 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:41.928698    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 01:05:42.688250 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:42.688213    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hhzdd_f7abbbc7-1072-4563-9ebd-1ebe917ebca5/discovery/0.log"
Nov 04 01:05:42.877325 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:42.877287    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-bqzd8_2354e993-f4ef-4b87-956b-b35a01cef507/kube-auth-proxy/0.log"
Nov 04 01:05:43.444238 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:43.444203    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-9t5h4_eb9de502-2a33-42ea-996d-f96b86c57a3f/istio-proxy/0.log"
Nov 04 01:05:44.641380 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:44.641328    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-sjtz5_8035f969-cba1-40f0-904d-ee65b610b96c/istio-proxy/0.log"
Nov 04 01:05:44.845011 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:44.844972    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-tcztg_c2ce493a-180f-47a6-a03e-74f3db88976f/istio-proxy/0.log"
Nov 04 01:05:45.034807 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:45.034712    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-x42sk_fb4b017e-aceb-4f7d-8470-4af3187aea6b/istio-proxy/0.log"
Nov 04 01:05:46.966650 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:46.966618    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-77vbh_bfcab315-9f9c-4214-9fd7-36811b39c65b/main/0.log"
Nov 04 01:05:47.345163 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:47.345132    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-w2wvx_9a6ecec9-7825-4edb-9cb2-4a38bca0a52c/main/0.log"
Nov 04 01:05:47.706735 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:47.706651    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bw6nj_87912bc2-fd56-4bbd-81f1-e31e390f0727/main/0.log"
Nov 04 01:05:48.095220 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:48.095182    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vdzxd_eff13fca-8c3c-483a-92aa-b8844a6712af/main/0.log"
Nov 04 01:05:48.280562 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:05:48.280520    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-k8xwd_21599918-1626-411b-8922-0270ebf1e061/main/0.log"
Nov 04 01:06:04.830361 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:06:04.830273    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:06:23.031478 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:23.031440    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fpnrk_d128c5e4-fb0c-486a-a7ef-fa8545746825/global-pull-secret-syncer/0.log"
Nov 04 01:06:23.513294 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:23.513255    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jnzl8_b0ddeb79-97e0-492a-a4d6-f50f96720fb7/konnectivity-agent/0.log"
Nov 04 01:06:23.623804 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:23.623770    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal_b6a44b6090d9fc279a15fb8a9a158ae7/haproxy/0.log"
Nov 04 01:06:29.155180 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:29.155136    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:06:31.756181 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:31.756138    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:06:34.395614 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:34.395575    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:06:34.831108 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:06:34.831069    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:06:40.937220 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:40.937182    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hhzdd_f7abbbc7-1072-4563-9ebd-1ebe917ebca5/discovery/0.log"
Nov 04 01:06:40.947196 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:40.947162    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-bqzd8_2354e993-f4ef-4b87-956b-b35a01cef507/kube-auth-proxy/0.log"
Nov 04 01:06:40.998435 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:40.998373    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-9t5h4_eb9de502-2a33-42ea-996d-f96b86c57a3f/istio-proxy/0.log"
Nov 04 01:06:41.109742 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:41.109708    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-sjtz5_8035f969-cba1-40f0-904d-ee65b610b96c/istio-proxy/0.log"
Nov 04 01:06:41.124851 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:41.124824    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-tcztg_c2ce493a-180f-47a6-a03e-74f3db88976f/istio-proxy/0.log"
Nov 04 01:06:41.141564 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:41.141531    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-x42sk_fb4b017e-aceb-4f7d-8470-4af3187aea6b/istio-proxy/0.log"
Nov 04 01:06:41.502438 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:41.502349    6430 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-east-2.compute.internal" podUID="b6a44b6090d9fc279a15fb8a9a158ae7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 04 01:06:44.718696 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:44.718660    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 01:06:44.837325 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:44.837289    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 01:06:44.853867 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:44.853840    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 01:06:44.924119 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:44.924085    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"
Nov 04 01:06:49.599518 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:49.599477    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 01:06:49.722937 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:49.722896    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 01:06:49.756897 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:49.756837    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 01:06:49.827602 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:49.827570    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"
Nov 04 01:06:50.704108 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.704075    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/alertmanager/0.log"
Nov 04 01:06:50.743069 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.743028    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/config-reloader/0.log"
Nov 04 01:06:50.772215 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.772188    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/kube-rbac-proxy-web/0.log"
Nov 04 01:06:50.801705 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.801680    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/kube-rbac-proxy/0.log"
Nov 04 01:06:50.831097 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.831071    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/kube-rbac-proxy-metric/0.log"
Nov 04 01:06:50.860834 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.860811    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/prom-label-proxy/0.log"
Nov 04 01:06:50.891074 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.891046    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a39f5e0e-2391-4581-ad5f-ec7544e32bdb/init-config-reloader/0.log"
Nov 04 01:06:50.995011 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:50.994907    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cjpmb_f178f2de-cbfa-41db-9568-c52a97e0ec2d/kube-state-metrics/0.log"
Nov 04 01:06:51.019254 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.019219    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cjpmb_f178f2de-cbfa-41db-9568-c52a97e0ec2d/kube-rbac-proxy-main/0.log"
Nov 04 01:06:51.049365 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.049339    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cjpmb_f178f2de-cbfa-41db-9568-c52a97e0ec2d/kube-rbac-proxy-self/0.log"
Nov 04 01:06:51.083439 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.083384    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b4f46b88d-r6wq8_89775551-7997-496e-84ac-55070f5b6665/metrics-server/0.log"
Nov 04 01:06:51.121874 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.121817    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-zchvm_5326b538-687b-4b39-b610-cf9b67362c02/monitoring-plugin/0.log"
Nov 04 01:06:51.152564 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.152533    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdd9_026dcdf5-84bf-47d3-a512-6eef0ae86f5c/node-exporter/0.log"
Nov 04 01:06:51.180734 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.180702    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdd9_026dcdf5-84bf-47d3-a512-6eef0ae86f5c/kube-rbac-proxy/0.log"
Nov 04 01:06:51.210766 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.210738    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cdd9_026dcdf5-84bf-47d3-a512-6eef0ae86f5c/init-textfile/0.log"
Nov 04 01:06:51.441337 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.441292    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ts8bj_10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/kube-rbac-proxy-main/0.log"
Nov 04 01:06:51.469790 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.469757    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ts8bj_10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/kube-rbac-proxy-self/0.log"
Nov 04 01:06:51.500481 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.500445    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ts8bj_10a9a502-a9d0-448d-ae12-e7fa44dc3c2f/openshift-state-metrics/0.log"
Nov 04 01:06:51.981547 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:51.981506    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/thanos-query/0.log"
Nov 04 01:06:52.009600 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:52.009572    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/kube-rbac-proxy-web/0.log"
Nov 04 01:06:52.042006 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:52.041979    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/kube-rbac-proxy/0.log"
Nov 04 01:06:52.071617 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:52.071581    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/prom-label-proxy/0.log"
Nov 04 01:06:52.099925 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:52.099895    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/kube-rbac-proxy-rules/0.log"
Nov 04 01:06:52.129492 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:52.129459    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8b55658f-66dwg_ecfc458d-4b78-4b41-8ea4-79514fa4fe0f/kube-rbac-proxy-metrics/0.log"
Nov 04 01:06:55.473024 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:55.472993    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/manager/1.log"
Nov 04 01:06:55.507574 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:55.507537    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/manager/0.log"
Nov 04 01:06:55.514901 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:55.514874    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/copy-manifests/0.log"
Nov 04 01:06:56.109296 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:56.109268    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ngzx9_0f71c198-ea98-4678-8297-3d93e18fc042/dns/0.log"
Nov 04 01:06:56.134147 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:56.134119    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ngzx9_0f71c198-ea98-4678-8297-3d93e18fc042/kube-rbac-proxy/0.log"
Nov 04 01:06:56.164769 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:56.164742    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-784mp_04000032-9127-46a5-ba82-dcbb72b1e394/dns-node-resolver/0.log"
Nov 04 01:06:56.993214 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:56.993183    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6rhdg_a1b695b8-84da-4df4-87f7-c3aa1ccdcf9f/node-ca/0.log"
Nov 04 01:06:58.003430 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:06:58.003366    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:07:04.840794 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:07:04.840739    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:07:05.701715 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:05.701680    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hhzdd_f7abbbc7-1072-4563-9ebd-1ebe917ebca5/discovery/0.log"
Nov 04 01:07:06.151638 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.151600    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-bqzd8_2354e993-f4ef-4b87-956b-b35a01cef507/kube-auth-proxy/0.log"
Nov 04 01:07:06.267173 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.267130    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-9t5h4_eb9de502-2a33-42ea-996d-f96b86c57a3f/istio-proxy/0.log"
Nov 04 01:07:06.413931 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.413845    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hhzdd_f7abbbc7-1072-4563-9ebd-1ebe917ebca5/discovery/0.log"
Nov 04 01:07:06.477524 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.477485    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-sjtz5_8035f969-cba1-40f0-904d-ee65b610b96c/istio-proxy/0.log"
Nov 04 01:07:06.494047 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.494023    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-bqzd8_2354e993-f4ef-4b87-956b-b35a01cef507/kube-auth-proxy/0.log"
Nov 04 01:07:06.510267 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.510234    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-tcztg_c2ce493a-180f-47a6-a03e-74f3db88976f/istio-proxy/0.log"
Nov 04 01:07:06.548188 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.548158    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-x42sk_fb4b017e-aceb-4f7d-8470-4af3187aea6b/istio-proxy/0.log"
Nov 04 01:07:06.550650 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.550616    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-9t5h4_eb9de502-2a33-42ea-996d-f96b86c57a3f/istio-proxy/0.log"
Nov 04 01:07:06.653251 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.653218    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-sjtz5_8035f969-cba1-40f0-904d-ee65b610b96c/istio-proxy/0.log"
Nov 04 01:07:06.668297 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.668225    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-tcztg_c2ce493a-180f-47a6-a03e-74f3db88976f/istio-proxy/0.log"
Nov 04 01:07:06.686293 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:06.686267    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-x42sk_fb4b017e-aceb-4f7d-8470-4af3187aea6b/istio-proxy/0.log"
Nov 04 01:07:08.928789 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:08.928752    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wqzj_b9ed813d-7b33-4539-8d6f-657a80430229/serve-healthcheck-canary/0.log"
Nov 04 01:07:09.678314 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:09.678277    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cswbt_a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/kube-rbac-proxy/0.log"
Nov 04 01:07:09.706280 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:09.706249    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cswbt_a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/exporter/0.log"
Nov 04 01:07:09.736241 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:09.736213    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cswbt_a378b9cf-dc33-42fa-8c6e-de8497ba0c2f/extractor/0.log"
Nov 04 01:07:11.387962 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.387920    6430 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"]
Nov 04 01:07:11.388519 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.388348    6430 cpu_manager.go:414] "containerMap: removing container" podUID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" containerName="maas-api"
Nov 04 01:07:11.388519 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.388359    6430 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" containerName="maas-api"
Nov 04 01:07:11.388519 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.388444    6430 memory_manager.go:365] "RemoveStaleState removing state" podUID="59db20c6-b9de-4de9-b16d-48a648fc7e8b" containerName="maas-api"
Nov 04 01:07:11.391708 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.391688    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.401516 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.401486    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"]
Nov 04 01:07:11.401626 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.401600    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"openshift-service-ca.crt\""
Nov 04 01:07:11.401669 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.401656    6430 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"kube-root-ca.crt\""
Nov 04 01:07:11.401669 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.401662    6430 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-595p9\"/\"default-dockercfg-whg7p\""
Nov 04 01:07:11.458572 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.458537    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-podres\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.458762 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.458598    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65f6\" (UniqueName: \"kubernetes.io/projected/401d8309-21d0-4629-8e37-353724367631-kube-api-access-d65f6\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.458762 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.458649    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-lib-modules\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.458841 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.458752    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-sys\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.458841 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.458813    6430 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-proc\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560173 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560136    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-podres\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560368 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560187    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65f6\" (UniqueName: \"kubernetes.io/projected/401d8309-21d0-4629-8e37-353724367631-kube-api-access-d65f6\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560368 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560267    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-podres\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560368 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560292    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-lib-modules\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560368 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560331    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-sys\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560541 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560380    6430 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-proc\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560541 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560452    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-sys\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560541 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560458    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-lib-modules\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.560541 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.560503    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/401d8309-21d0-4629-8e37-353724367631-proc\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.574213 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.574178    6430 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65f6\" (UniqueName: \"kubernetes.io/projected/401d8309-21d0-4629-8e37-353724367631-kube-api-access-d65f6\") pod \"perf-node-gather-daemonset-jlcb4\" (UID: \"401d8309-21d0-4629-8e37-353724367631\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.706917 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.706816    6430 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:11.983344 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:11.983237    6430 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"]
Nov 04 01:07:11.986319 ip-10-0-143-134 kubenswrapper[6430]: W1104 01:07:11.986285    6430 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod401d8309_21d0_4629_8e37_353724367631.slice/crio-0683bbe5d3c119044bfb05b646a2c4caee0c6a1d6846ad9a10f3d221b4eafd54 WatchSource:0}: Error finding container 0683bbe5d3c119044bfb05b646a2c4caee0c6a1d6846ad9a10f3d221b4eafd54: Status 404 returned error can't find the container with id 0683bbe5d3c119044bfb05b646a2c4caee0c6a1d6846ad9a10f3d221b4eafd54
Nov 04 01:07:12.192943 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.192910    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/manager/1.log"
Nov 04 01:07:12.227491 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.227455    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/manager/0.log"
Nov 04 01:07:12.250959 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.250883    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-lhbhz_82a246af-db2d-4e41-9c03-a36945082ffd/copy-manifests/0.log"
Nov 04 01:07:12.585001 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.584963    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4" event={"ID":"401d8309-21d0-4629-8e37-353724367631","Type":"ContainerStarted","Data":"cb6bf04aaf3b68078b07ddcae3b95c0aa8efaabdfdcb6b73dbf6320416c1f3bf"}
Nov 04 01:07:12.585454 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.585028    6430 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4" event={"ID":"401d8309-21d0-4629-8e37-353724367631","Type":"ContainerStarted","Data":"0683bbe5d3c119044bfb05b646a2c4caee0c6a1d6846ad9a10f3d221b4eafd54"}
Nov 04 01:07:12.585454 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.585056    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:12.607071 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:12.607016    6430 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4" podStartSLOduration=1.607001762 podStartE2EDuration="1.607001762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:12.603968191 +0000 UTC m=+5848.495030120" watchObservedRunningTime="2026-11-04 01:07:12.607001762 +0000 UTC m=+5848.498063696"
Nov 04 01:07:16.146788 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:16.146744    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vssxc_2ad3722d-01d2-41f6-84f4-9e16b92c0060/manager/0.log"
Nov 04 01:07:18.599949 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:18.599916    6430 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-jlcb4"
Nov 04 01:07:19.893479 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:19.893443    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 01:07:19.961772 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:19.961739    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 01:07:19.990028 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:19.989995    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 01:07:20.058553 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:20.058519    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"
Nov 04 01:07:23.575686 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.575612    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/kube-multus-additional-cni-plugins/0.log"
Nov 04 01:07:23.609263 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.609235    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/egress-router-binary-copy/0.log"
Nov 04 01:07:23.677286 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.677253    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/cni-plugins/0.log"
Nov 04 01:07:23.768584 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.768555    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/bond-cni-plugin/0.log"
Nov 04 01:07:23.809319 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.809288    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/routeoverride-cni/0.log"
Nov 04 01:07:23.853380 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.853293    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/whereabouts-cni-bincopy/0.log"
Nov 04 01:07:23.893033 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:23.892995    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hm9dt_04b2245e-79d5-4117-a5e1-2dfc4ee0e5c3/whereabouts-cni/0.log"
Nov 04 01:07:24.497700 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:24.497665    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-llrvd_5eeb5df1-2d70-4963-ab3f-503518320882/kube-multus/0.log"
Nov 04 01:07:24.846802 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:24.846762    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lz9rf_bcfd05a2-2670-445f-8d81-412fa24a1d60/network-metrics-daemon/0.log"
Nov 04 01:07:24.869934 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:24.869897    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lz9rf_bcfd05a2-2670-445f-8d81-412fa24a1d60/kube-rbac-proxy/0.log"
Nov 04 01:07:26.989834 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:26.989796    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/ovn-controller/0.log"
Nov 04 01:07:27.012963 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.012924    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/ovn-acl-logging/0.log"
Nov 04 01:07:27.043966 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.043927    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/kube-rbac-proxy-node/0.log"
Nov 04 01:07:27.074212 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.074159    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/kube-rbac-proxy-ovn-metrics/0.log"
Nov 04 01:07:27.105354 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.105322    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/northd/0.log"
Nov 04 01:07:27.134030 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.133998    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/nbdb/0.log"
Nov 04 01:07:27.167361 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.167327    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/sbdb/0.log"
Nov 04 01:07:27.467166 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:27.467132    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qts9p_c17332ae-befe-4495-8904-af7ad229d464/ovnkube-controller/0.log"
Nov 04 01:07:28.928147 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:28.928113    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zr4h4_6b40159c-efef-4f32-91d3-7f0d6a74a146/network-check-target-container/0.log"
Nov 04 01:07:30.377336 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:30.377298    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wf22x_9570168c-b9aa-455f-981e-33d57c4b5671/iptables-alerter/0.log"
Nov 04 01:07:31.343533 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:31.343498    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-ffxlw_5a32217f-0822-43ae-8541-066002fca229/tuned/0.log"
Nov 04 01:07:34.831771 ip-10-0-143-134 kubenswrapper[6430]: E1104 01:07:34.831726    6430 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cswbt" containerName="extractor"
Nov 04 01:07:35.916759 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:35.916720    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn8cg_0e628017-63f5-4329-86b0-f07e84304f96/csi-driver/0.log"
Nov 04 01:07:35.946176 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:35.946148    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn8cg_0e628017-63f5-4329-86b0-f07e84304f96/csi-node-driver-registrar/0.log"
Nov 04 01:07:35.975280 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:35.975250    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn8cg_0e628017-63f5-4329-86b0-f07e84304f96/csi-liveness-probe/0.log"
Nov 04 01:07:50.580127 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:50.580092    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9szhc_9633fbd0-0587-402d-bdd0-ea2072eaeee5/manager/0.log"
Nov 04 01:07:50.650052 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:50.650021    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jt564_84f95bcd-d2d7-481a-b499-3458acd41ddd/limitador/0.log"
Nov 04 01:07:50.668397 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:50.668360    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/1.log"
Nov 04 01:07:50.737576 ip-10-0-143-134 kubenswrapper[6430]: I1104 01:07:50.737542    6430 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rhgs4_c8a4965f-af3d-4332-a17a-92947cef5216/manager/2.log"